1 Commits
Author SHA1 Message Date
sneak 88e0c6c025 Bound each shutdown stage by the stop deadline (closes #100)
check / check (push) Canceled after 0s
Shutdown passes its context to every stage: the streamer, the four
handlers, the WHOIS fetcher, the database maintainer and the HTTP server
each stop waiting when it ends. A handler then skips its last write and
logs how many updates it did not write, and the HTTP server closes the
connections still open. The database is checkpointed and closed last,
and left open once the deadline has passed, since a write may still
hold it. The stop hook gives Shutdown 8 seconds and fx 9, inside the 10
seconds upaas and docker stop allow, so a stop signal exits 0.

Model: opus-5-5
2026-10-19 23:42:06 +00:00
37 changed files with 257 additions and 2022 deletions
+2 -12
View File
@@ -73,8 +73,7 @@
# This repository's host-built artifacts: `make build` writes bin/, `go
# build` in the root the binaries, `make run` log.txt, `make asupdate` its
# .tmp file; logs, test binaries, coverage profiles, source archives, and
# the binaries and libraries a build for another platform writes.
# .tmp file; test binaries, coverage profiles and source archives.
/bin
/routewatch
/asinfo-gen
@@ -83,24 +82,15 @@
/out
/pkg/asinfo/asdata.json
**/*.tar.zst
**/*.log
**/*.test
**/*.out
**/*.tmp
**/*.exe
**/*.dll
**/*.so
**/*.dylib
# Local databases. The image carries a source archive of the whole build
# context, so these would otherwise ship inside it. The daemon's database
# is db.sqlite, with its -wal and -shm files while it runs.
# context, so these would otherwise ship inside it.
**/*.db
**/*.db-journal
**/*.db-wal
**/*.sqlite
**/*.sqlite-wal
**/*.sqlite-shm
# A local Go workspace points at directories outside the build context.
/go.work
+1 -5
View File
@@ -67,14 +67,10 @@ node_modules/
go.work
go.work.sum
# Local databases, temporary files and debug output. The daemon's
# database is db.sqlite, with its -wal and -shm files while it runs.
# Local databases, temporary files and debug output.
*.db
*.db-journal
*.db-wal
*.sqlite
*.sqlite-wal
*.sqlite-shm
*.tmp
log.txt
out
+11 -38
View File
@@ -5,22 +5,6 @@ table in real time: it streams BGP UPDATE messages from the RIPE RIS Live
service, maintains a live routing table in SQLite, and provides HTTP APIs for
querying routing information.
## Rationale
RIPE RIS Live streams the BGP updates that RIPE's route collectors receive, as
they happen, from the moment a client connects. It reports changes, not the
routing table they add up to, so it cannot say which AS an IP address is routed
to now or which routes a prefix has. RouteWatch applies the feed's updates to
live routing tables in its own SQLite database and answers IP-to-AS lookups and
prefix pages from them.
The feed sends no table when a client connects, so these tables hold only the
routes the feed has announced since RouteWatch first started: an IP-to-AS lookup
finds nothing for an address whose routes have not been announced since then.
The tables also miss the updates RouteWatch drops when it falls behind under
load and those sent while it is stopped, so a lookup or a prefix page can show a
route the feed has withdrawn.
## Features
- Real-time streaming of BGP updates from RIPE RIS Live
@@ -52,8 +36,6 @@ DEBUG=routewatch ./routewatch
## HTTP Endpoints
Every route answers `HEAD` as it answers `GET`, without the body.
### Web Interface
- `GET /` - Home page (HTML): headline figures, and lookups by AS number and by
@@ -82,16 +64,11 @@ Every route answers `HEAD` as it answers `GET`, without the body.
collector, stream and handler queue statistics
- `GET /api/v1/ip/{ip}` - Looks up each address: its most specific live route
(`netblock`), that route's origin AS with its WHOIS details, and the address's
PTR names. `{ip}` is an IP address, a host name, or a comma-separated list of
at most 10 of these. Of the addresses a host name resolves to, only public
ones are looked up: a name with none, such as one that resolves to a private
address, is reported as `host name has no public address`, without its
addresses. `results` has one entry per address with a live route and `errors`
one per address or name that failed. The answer is 400 when `{ip}` lists more
than 10, when it comes to more than 10 addresses once its names are resolved,
or when nothing in it is an IP address or a host name with a public address;
and 404 when none of the addresses has a live route. A PTR lookup gets 2
seconds; an address whose lookup takes longer is answered without PTR names.
PTR names. `{ip}` is an IP address, a host name (each address it resolves to
is looked up), or a comma-separated list of these. `results` has one entry per
address with a live route and `errors` one per address or name that failed.
The answer is 400 when nothing in `{ip}` is an IP address or a host name that
resolves, and 404 when none of the addresses has a live route.
- `GET /api/v1/as/{asn}` - The AS and the prefixes it originates, IPv4 and IPv6
apart, each with when its newest route was updated
- `GET /api/v1/prefix/{prefix}/{length}` - Every live route for the prefix, one
@@ -108,11 +85,8 @@ and 4 seconds for the database reads behind `/status.json` and `/api/v1/stats`.
An AS number, prefix or mask length that does not parse answers 400, as does a
mask length out of range for its IP version. An AS the database does not hold,
or a prefix with no live route, answers 404. The HTML pages give these errors as
plain text. Under `/api/v1` and `/ip/`, a path no route matches answers 404 with
the message `Not found`, and `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`,
`TRACE` or `CONNECT` answers 405 with `Method not allowed`. A 500 answer says
only `Internal server error`, and a host name that does not resolve only
`could not resolve host name`; the detail goes to the log.
plain text. A 500 answer says only `Internal server error`, and a host name that
does not resolve only `could not resolve host name`; the detail goes to the log.
Every answer carries `Strict-Transport-Security`, `Content-Security-Policy`,
`X-Frame-Options`, `X-Content-Type-Options`, `Referrer-Policy` and
@@ -211,8 +185,7 @@ routewatch/
- **ASHandler**: Tracks all ASNs seen in AS paths
- **PeerHandler**: Records BGP peer information
- **PrefixHandler**: Maintains live routing table (upserts on announcement,
deletes on withdrawal or on an announcement whose AS path is empty or
holds AS 0 anywhere, deletes all of a peer's routes when its BGP session
deletes on withdrawal, deletes all of a peer's routes when its BGP session
goes down). Routes are kept per peer IP, so when one peer IP has sessions
on two collectors, a session going down on one also removes the routes the
other still carries, until they are announced again.
@@ -231,9 +204,9 @@ routewatch/
capacity
- **Graceful Shutdown**: on SIGTERM or SIGINT the streamer stops, the handlers
write the batches they hold, the HTTP server stops and the database is
checkpointed and closed. The daemon gives this 8 seconds and exits with status
0; "Running under upaas" says what a stop loses, logs and leaves undone when
the 8 seconds run out.
checkpointed and closed, all within 8 seconds, and the daemon exits with
status 0. What is not written by then is lost, and the handler logs how many
updates (see "Running under upaas").
- **Reconnection**: Any end of the stream other than shutdown, including RIS
Live ending the response normally, marks the feed disconnected and reconnects
with exponential backoff (5s-320s). The backoff resets after a stream that
+4 -9
View File
@@ -148,9 +148,7 @@ func TestLiveCountsTrackWritesInRealtime(t *testing.T) {
func writeASNsPrefixesAndPeers(t *testing.T, db *Database, ts time.Time) {
t.Helper()
seen := SeenTimes{First: ts, Last: ts}
err := db.GetOrCreateASNBatch(map[int]SeenTimes{64500: seen, 64501: seen})
err := db.GetOrCreateASNBatch(map[int]time.Time{64500: ts, 64501: ts})
if err != nil {
t.Fatalf("GetOrCreateASNBatch: %v", err)
}
@@ -166,14 +164,14 @@ func writeASNsPrefixesAndPeers(t *testing.T, db *Database, ts time.Time) {
err = db.UpdatePeerBatch(map[string]PeerUpdate{
testV4PeerIP: {
PeerIP: testV4PeerIP, PeerASN: 64500,
MessageType: "UPDATE", Seen: seen,
MessageType: "UPDATE", Timestamp: ts,
},
})
if err != nil {
t.Fatalf("UpdatePeerBatch: %v", err)
}
err = db.RecordPeering(64500, 64501, seen)
err = db.RecordPeering(64500, 64501, ts)
if err != nil {
t.Fatalf("RecordPeering: %v", err)
}
@@ -194,10 +192,7 @@ func TestLiveCountsSeededFromDatabaseAtStartup(t *testing.T) {
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
seen := SeenTimes{First: ts, Last: ts}
err = db.GetOrCreateASNBatch(
map[int]SeenTimes{64500: seen, 64501: seen, 64502: seen})
err = db.GetOrCreateASNBatch(map[int]time.Time{64500: ts, 64501: ts, 64502: ts})
if err != nil {
t.Fatalf("GetOrCreateASNBatch: %v", err)
}
+31 -73
View File
@@ -169,20 +169,6 @@ func New(cfg *config.Config, logger *logger.Logger) (*Database, error) {
return nil, fmt.Errorf("failed to create database directory: %w", err)
}
// The size of the WAL file the previous run left, which Initialize logs
// with the start-up checkpoint. It is read before the database is opened,
// because opening it writes a frame to the WAL. There is no WAL file when
// the database is new or the last run closed the database, which deletes
// it.
var walBytes int64
wal, err := os.Stat(dbPath + walFileSuffix)
if err == nil {
walBytes = wal.Size()
} else if !errors.Is(err, os.ErrNotExist) {
logger.Warn("Failed to read the WAL file's size on startup", "error", err)
}
// Per-connection SQLite settings go in the DSN so every pooled connection
// gets them, not just the one that runs the Initialize pragmas. _txlock=
// immediate makes every transaction take the write lock at BEGIN. Without it
@@ -243,7 +229,7 @@ func New(cfg *config.Config, logger *logger.Logger) (*Database, error) {
counts: &liveCounts{},
}
err = database.Initialize(walBytes)
err = database.Initialize()
if err != nil {
return nil, fmt.Errorf("failed to initialize database: %w", err)
}
@@ -259,16 +245,8 @@ func New(cfg *config.Config, logger *logger.Logger) (*Database, error) {
return database, nil
}
// The messages Initialize logs the start-up WAL checkpoint with.
const (
startupCheckpointMessage = "WAL checkpoint on startup"
startupCheckpointIncompleteMessage = "WAL checkpoint on startup incomplete"
)
// Initialize creates the database schema if it doesn't exist. walBytes is the
// size of the WAL file the previous run left, which New reads before it opens
// the database.
func (d *Database) Initialize(walBytes int64) error {
// Initialize creates the database schema if it doesn't exist.
func (d *Database) Initialize() error {
// Set SQLite pragmas for performance. Per-connection settings (cache_size,
// synchronous, busy_timeout, journal_mode) live in the DSN, and New sets the
// writing connection's cache_size; temp_store is left at its default so
@@ -288,33 +266,19 @@ func (d *Database) Initialize(walBytes int64) error {
}
}
// The start-up checkpoint copies the WAL a previous run left into the
// database file and empties the WAL file; its frame counts then read 0,
// so walBytes is logged with them.
var busy, walFrames, checkpointedFrames int
// Run WAL checkpoint on startup to consolidate any existing WAL
var walPages, checkpointedPages, movedPages int
err := d.writer.QueryRowContext(context.Background(),
"PRAGMA wal_checkpoint(TRUNCATE)",
).Scan(&busy, &walFrames, &checkpointedFrames)
switch {
case err != nil:
).Scan(&walPages, &checkpointedPages, &movedPages)
if err != nil {
d.logger.Warn("Failed to checkpoint WAL on startup", "error", err)
case busy != 0:
// Another process using the database kept the checkpoint from
// copying or emptying all of the WAL.
d.logger.Warn(startupCheckpointIncompleteMessage,
"wal_bytes", walBytes,
"busy", busy,
"wal_frames", walFrames,
"checkpointed_frames", checkpointedFrames,
)
default:
d.logger.Info(startupCheckpointMessage,
"wal_bytes", walBytes,
"busy", busy,
"wal_frames", walFrames,
"checkpointed_frames", checkpointedFrames,
} else {
d.logger.Info("WAL checkpoint on startup",
"wal_pages", walPages,
"checkpointed", checkpointedPages,
"moved", movedPages,
)
}
@@ -891,7 +855,7 @@ func insertPrefixesInTable(
// UpdatePrefixesBatch stores, in a single transaction, the prefixes that are
// not stored yet, each first seen at its time in prefixes: the IPv4 prefixes
// first and then the IPv6 prefixes. A prefix that does not parse is not stored.
// first and then the IPv6 prefixes.
func (d *Database) UpdatePrefixesBatch(prefixes map[string]time.Time) error {
if len(prefixes) == 0 {
return nil
@@ -910,11 +874,6 @@ func (d *Database) UpdatePrefixesBatch(prefixes map[string]time.Time) error {
prefixesV6 := make(map[string]time.Time)
for prefix, timestamp := range prefixes {
_, err := netip.ParsePrefix(prefix)
if err != nil {
continue
}
if detectIPVersion(prefix) == ipVersionV4 {
prefixesV4[prefix] = timestamp
} else {
@@ -944,14 +903,13 @@ func (d *Database) UpdatePrefixesBatch(prefixes map[string]time.Time) error {
return nil
}
// newASN returns a new ASN record first seen at seen.First and last seen at
// seen.Last, with the handle and description from the AS info lookup when it
// knows the number.
func newASN(number int, seen SeenTimes) ASN {
// newASN returns a new ASN record first and last seen at timestamp, with the
// handle and description from the AS info lookup when it knows the number.
func newASN(number int, timestamp time.Time) ASN {
asn := ASN{
ASN: number,
FirstSeen: seen.First,
LastSeen: seen.Last,
FirstSeen: timestamp,
LastSeen: timestamp,
}
// Look up ASN info
@@ -963,12 +921,12 @@ func newASN(number int, seen SeenTimes) ASN {
return asn
}
// upsertASN sets the last_seen time of ASN number to seen.Last with the given
// upsertASN updates the last_seen time of ASN number with the given
// statements, inserting the ASN when it is new. It reports whether it inserted
// it.
func upsertASN(
ctx context.Context, selectStmt, updateStmt, insertStmt *sql.Stmt,
number int, seen SeenTimes,
number int, timestamp time.Time,
) (bool, error) {
var (
asn ASN
@@ -979,7 +937,7 @@ func upsertASN(
&description, &asn.FirstSeen, &asn.LastSeen)
if err == nil {
// ASN exists, update last_seen
_, err = updateStmt.ExecContext(ctx, seen.Last, number)
_, err = updateStmt.ExecContext(ctx, timestamp, number)
if err != nil {
return false, fmt.Errorf("failed to update ASN %d: %w", number, err)
}
@@ -992,7 +950,7 @@ func upsertASN(
}
// ASN doesn't exist, create it
asn = newASN(number, seen)
asn = newASN(number, timestamp)
_, err = insertStmt.ExecContext(ctx, asn.ASN, asn.Handle,
asn.Description, asn.FirstSeen, asn.LastSeen)
@@ -1004,7 +962,7 @@ func upsertASN(
}
// GetOrCreateASNBatch creates or updates multiple ASNs in a single transaction
func (d *Database) GetOrCreateASNBatch(asns map[int]SeenTimes) error {
func (d *Database) GetOrCreateASNBatch(asns map[int]time.Time) error {
if len(asns) == 0 {
return nil
}
@@ -1046,9 +1004,9 @@ func (d *Database) GetOrCreateASNBatch(asns map[int]SeenTimes) error {
var newASNs int
for number, seen := range asns {
for number, timestamp := range asns {
inserted, err := upsertASN(ctx, selectStmt, updateStmt, insertStmt,
number, seen)
number, timestamp)
if err != nil {
return err
}
@@ -1117,7 +1075,7 @@ func (d *Database) GetOrCreateASN(number int, timestamp time.Time) (*ASN, error)
}
// ASN doesn't exist, create it with ASN info lookup
asn = newASN(number, SeenTimes{First: timestamp, Last: timestamp})
asn = newASN(number, timestamp)
_, err = tx.Exec("INSERT INTO asns "+
"(asn, handle, description, first_seen, last_seen) "+
@@ -1224,7 +1182,7 @@ func (d *Database) RecordAnnouncement(announcement *Announcement) error {
}
// RecordPeering records a peering relationship between two ASNs.
func (d *Database) RecordPeering(asA, asB int, seen SeenTimes) error {
func (d *Database) RecordPeering(asA, asB int, timestamp time.Time) error {
// Validate ASNs
if asA <= 0 || asB <= 0 {
return fmt.Errorf("%w: asA=%d, asB=%d", errInvalidASN, asA, asB)
@@ -1260,12 +1218,12 @@ func (d *Database) RecordPeering(asA, asB int, seen SeenTimes) error {
if exists {
_, err = tx.Exec(
"UPDATE peerings SET last_seen = ? WHERE as_a = ? AND as_b = ?",
seen.Last, asA, asB)
timestamp, asA, asB)
} else {
_, err = tx.Exec(`
INSERT INTO peerings (id, as_a, as_b, first_seen, last_seen)
VALUES (?, ?, ?, ?, ?)`,
generateUUID().String(), asA, asB, seen.First, seen.Last)
generateUUID().String(), asA, asB, timestamp, timestamp)
}
if err != nil {
@@ -1343,11 +1301,11 @@ func (d *Database) UpdatePeerBatch(peers map[string]PeerUpdate) error {
if exists {
_, err = updateStmt.ExecContext(ctx, update.PeerASN,
update.Seen.Last, update.MessageType, update.PeerIP)
update.Timestamp, update.MessageType, update.PeerIP)
} else {
_, err = insertStmt.ExecContext(ctx,
generateUUID().String(), update.PeerIP, update.PeerASN,
update.Seen.First, update.Seen.Last, update.MessageType)
update.Timestamp, update.Timestamp, update.MessageType)
}
if err != nil {
+4 -221
View File
@@ -1,10 +1,8 @@
package database
import (
"bytes"
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"log/slog"
@@ -617,12 +615,10 @@ func TestBatchWriteDuringCheckpoint(t *testing.T) {
})
ts := time.Now().UTC()
seen := SeenTimes{First: ts, Last: ts}
for i := range contentionIterations {
asns := map[int]SeenTimes{
i % contendedASNCount: seen,
(i % contendedASNCount) + asnSecondBand: seen,
asns := map[int]time.Time{
i % contendedASNCount: ts,
(i % contendedASNCount) + asnSecondBand: ts,
}
err := db.GetOrCreateASNBatch(asns)
@@ -751,8 +747,7 @@ func TestBatchWriteWaitsForAnalyze(t *testing.T) {
db, messages := newDatabaseWithLockMessages(t)
ctx := context.Background()
now := time.Now().UTC()
asns := map[int]SeenTimes{lockTestASN: {First: now, Last: now}}
asns := map[int]time.Time{lockTestASN: time.Now().UTC()}
// A stored AS gives ANALYZE a table to analyze and statistics to write.
err := db.GetOrCreateASNBatch(asns)
@@ -1054,176 +1049,6 @@ func walFileSize(t *testing.T, db *Database) int64 {
return wal.Size()
}
// TestStartupCheckpointLogsItsFigures opens a new database, writes a route,
// and opens the same file again while the first stays open, so that its WAL
// file stays. The first open finds no WAL file: it logs a size of 0 and no
// warning. The second logs the size of the WAL file the first left, and the
// checkpoint's busy flag and frame counts, each under its own name; the
// counts read 0 once the checkpoint has emptied the file.
// Initialize logged the busy flag as wal_pages and the counts under each
// other's names (https://git.eeqj.de/sneak/routewatch/issues/117).
func TestStartupCheckpointLogsItsFigures(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()}
db, lines := openLogged(t, cfg)
for _, line := range lines {
if line[slog.LevelKey] == slog.LevelWarn.String() {
t.Errorf("opening a new database logged a warning: %v", line)
}
}
line := findLine(t, lines, startupCheckpointMessage)
if line["wal_bytes"] != 0.0 {
t.Errorf("new database: start-up checkpoint logged wal_bytes = %v, "+
"want 0", line["wal_bytes"])
}
err := db.UpsertLiveRouteBatch([]*LiveRoute{
mkV4Route(t, "192.0.2.0/24", lockTestASN, time.Now().UTC()),
})
if err != nil {
t.Fatalf("failed to store a route: %v", err)
}
walBytes := walFileSize(t, db)
if walBytes == 0 {
t.Fatal("WAL file empty after the route write, want frames in it")
}
_, lines = openLogged(t, cfg)
line = findLine(t, lines, startupCheckpointMessage)
for _, key := range []string{"busy", "wal_frames", "checkpointed_frames"} {
if line[key] != 0.0 {
t.Errorf("start-up checkpoint logged %s = %v, want 0", key, line[key])
}
}
if line["wal_bytes"] != float64(walBytes) {
t.Errorf("start-up checkpoint logged wal_bytes = %v, want the %d "+
"bytes the first database left", line["wal_bytes"], walBytes)
}
}
// TestStartupCheckpointWarnsWhenIncomplete keeps a read open on a database
// while a route is written after the read began, then opens the same file
// again. The second open's start-up checkpoint cannot copy the frames written
// after the read began, and gives up once the busy timeout has passed. It
// checks that the checkpoint is logged as a warning with busy 1 and fewer
// frames copied than the WAL holds.
func TestStartupCheckpointWarnsWhenIncomplete(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()}
db, _ := openLogged(t, cfg)
ctx := context.Background()
reader, err := db.db.Conn(ctx)
if err != nil {
t.Fatalf("failed to open the reading connection: %v", err)
}
defer func() { _ = reader.Close() }()
// A plain BEGIN starts a transaction that only reads, unlike the BEGIN
// IMMEDIATE of _txlock. It keeps the snapshot its first read takes until
// it ends.
_, err = reader.ExecContext(ctx, "BEGIN")
if err != nil {
t.Fatalf("failed to begin the read transaction: %v", err)
}
defer func() { _, _ = reader.ExecContext(ctx, "ROLLBACK") }()
var routes int
err = reader.QueryRowContext(ctx,
"SELECT count(*) FROM live_routes_v4").Scan(&routes)
if err != nil {
t.Fatalf("failed to read the live routes: %v", err)
}
err = db.UpsertLiveRouteBatch([]*LiveRoute{
mkV4Route(t, "192.0.2.0/24", lockTestASN, time.Now().UTC()),
})
if err != nil {
t.Fatalf("failed to store a route after the read: %v", err)
}
_, lines := openLogged(t, cfg)
line := findLine(t, lines, startupCheckpointIncompleteMessage)
if line[slog.LevelKey] != slog.LevelWarn.String() {
t.Errorf("start-up checkpoint logged at level %v, want %s",
line[slog.LevelKey], slog.LevelWarn)
}
if line["busy"] != 1.0 {
t.Errorf("start-up checkpoint logged busy = %v, want 1", line["busy"])
}
walFrames, _ := line["wal_frames"].(float64)
checkpointedFrames, _ := line["checkpointed_frames"].(float64)
if checkpointedFrames >= walFrames {
t.Errorf("start-up checkpoint logged checkpointed_frames = %v and "+
"wal_frames = %v with a read open, want fewer copied than the WAL "+
"holds", line["checkpointed_frames"], line["wal_frames"])
}
}
// openLogged opens the database of cfg with a logger that writes JSON, and
// returns it with the lines it logged while it opened, decoded.
func openLogged(t *testing.T, cfg *config.Config) (*Database, []map[string]any) {
t.Helper()
var logged bytes.Buffer
db, err := New(cfg,
&logger.Logger{Logger: slog.New(slog.NewJSONHandler(&logged, nil))})
if err != nil {
t.Fatalf("failed to open the database: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
var lines []map[string]any
decoder := json.NewDecoder(&logged)
for decoder.More() {
var line map[string]any
err := decoder.Decode(&line)
if err != nil {
t.Fatalf("failed to decode the log: %v", err)
}
lines = append(lines, line)
}
return db, lines
}
// findLine returns the first of lines logged with message.
func findLine(
t *testing.T,
lines []map[string]any,
message string,
) map[string]any {
t.Helper()
for _, line := range lines {
if line[slog.MessageKey] == message {
return line
}
}
t.Fatalf("no line %q in the log", message)
return nil
}
// TestGetRandomPrefixesByLengthListsPrefixOnce stores one prefix of each
// family learned from two peers in the same second and checks that the
// prefix-length sample lists each of them once.
@@ -1838,48 +1663,6 @@ func TestUpdatePrefixesBatchKeepsFirstSeen(t *testing.T) {
}
}
// TestUpdatePeerBatchStoresTheEarliestAndLatestTimesOfANewPeer writes a new
// peer whose updates span 5 seconds. Its row is first seen at the earliest time
// and last seen at the latest.
func TestUpdatePeerBatchStoresTheEarliestAndLatestTimesOfANewPeer(
t *testing.T,
) {
t.Parallel()
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
first := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
last := first.Add(5 * time.Second)
err = db.UpdatePeerBatch(map[string]PeerUpdate{
testPeerIP: {
PeerIP: testPeerIP, PeerASN: 64500, MessageType: "UPDATE",
Seen: SeenTimes{First: first, Last: last},
},
})
if err != nil {
t.Fatalf("UpdatePeerBatch: %v", err)
}
var firstSeen, lastSeen time.Time
err = db.db.QueryRowContext(t.Context(),
"SELECT first_seen, last_seen FROM bgp_peers WHERE peer_ip = ?",
testPeerIP).Scan(&firstSeen, &lastSeen)
if err != nil {
t.Fatalf("reading peer %s: %v", testPeerIP, err)
}
if !firstSeen.Equal(first) || !lastSeen.Equal(last) {
t.Errorf("peer %s: first seen %s, last seen %s; want %s, %s",
testPeerIP, firstSeen, lastSeen, first, last)
}
}
// TestUpsertLiveRouteBatchWritesInKeyOrder writes new routes of two peers out
// of key order, the last one a second announcement of the first, and checks
// that the rows were inserted in key order, peer then prefix (each new row's id
@@ -1,61 +0,0 @@
package database
import (
"os"
"path/filepath"
"strings"
"testing"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/logger"
)
// TestDockerignoreExcludesDatabaseFiles checks that .dockerignore keeps the
// database file, and the WAL and shared-memory files SQLite writes next to it,
// out of the build context wherever they sit in the clone. The image ships the
// whole build context as a source archive.
func TestDockerignoreExcludesDatabaseFiles(t *testing.T) {
t.Parallel()
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
data, err := os.ReadFile("../../.dockerignore")
if err != nil {
t.Fatalf("failed to read .dockerignore: %v", err)
}
// Docker reads `**/` as any number of directories and the rest of the
// pattern as filepath.Match does, so `**/` followed by a pattern that
// matches a file's name excludes that file at any depth.
var anyDepth []string
for line := range strings.Lines(string(data)) {
pattern, ok := strings.CutPrefix(strings.TrimSpace(line), "**/")
if ok {
anyDepth = append(anyDepth, pattern)
}
}
name := filepath.Base(db.path)
for _, file := range []string{name, name + "-wal", name + "-shm"} {
excluded := false
for _, pattern := range anyDepth {
matched, err := filepath.Match(pattern, file)
if err != nil {
t.Fatalf("bad .dockerignore pattern **/%s: %v", pattern, err)
}
excluded = excluded || matched
}
if !excluded {
t.Errorf(".dockerignore has no **/ pattern that excludes %s", file)
}
}
}
+2 -2
View File
@@ -34,7 +34,7 @@ type Stats struct {
type Store interface {
// ASN operations
GetOrCreateASN(number int, timestamp time.Time) (*ASN, error)
GetOrCreateASNBatch(asns map[int]SeenTimes) error
GetOrCreateASNBatch(asns map[int]time.Time) error
// Prefix operations
GetOrCreatePrefix(prefix string, timestamp time.Time) (*Prefix, error)
@@ -44,7 +44,7 @@ type Store interface {
RecordAnnouncement(announcement *Announcement) error
// Peering operations
RecordPeering(asA, asB int, seen SeenTimes) error
RecordPeering(asA, asB int, timestamp time.Time) error
// Statistics
GetStats() (Stats, error)
+2 -25
View File
@@ -122,35 +122,12 @@ type LiveRouteDeletion struct {
IPVersion int
}
// SeenTimes is the earliest and the latest time one batch of updates saw an AS,
// a peer or a peering. A row the batch adds is stored first seen at First and
// last seen at Last; a row already stored keeps its first_seen and has its
// last_seen set to Last.
type SeenTimes struct {
First time.Time
Last time.Time
}
// Add makes t the earliest time of s when it is earlier than First, and the
// latest when it is later than Last. The zero SeenTimes holds no time yet, so
// the first Add sets both.
func (s *SeenTimes) Add(t time.Time) {
if s.First.IsZero() || t.Before(s.First) {
s.First = t
}
if s.Last.IsZero() || t.After(s.Last) {
s.Last = t
}
}
// PeerUpdate represents parameters for updating a peer. PeerASN and
// MessageType are those of the peer's latest update.
// PeerUpdate represents parameters for updating a peer
type PeerUpdate struct {
PeerIP string
PeerASN int
MessageType string
Seen SeenTimes
Timestamp time.Time
}
// ASNWHOISUpdate contains WHOIS data for updating an ASN record.
+9 -27
View File
@@ -4,7 +4,6 @@ package ristypes
import (
"encoding/json"
"slices"
"time"
)
@@ -25,8 +24,7 @@ const (
)
// ASPath is a BGP AS path as the feed writes it: an array of AS numbers in
// which an AS set is an array of its members, such as [1, 2, [3, 4]]. A path
// that holds AS 0 decodes as an empty one (UnmarshalJSON).
// which an AS set is an array of its members, such as [1, 2, [3, 4]].
type ASPath struct {
// Sequence is the path's AS numbers in order. A set of one member that
// ends the path counts as that AS and is its last element. The members of
@@ -50,41 +48,23 @@ func (p *ASPath) Origin() int {
return p.Sequence[len(p.Sequence)-1]
}
// UnmarshalJSON implements the json.Unmarshaler interface for ASPath. A path
// that holds AS 0 anywhere, a member of an AS set included, decodes as an empty
// path: RFC 7607 makes such a path malformed and its announcements
// withdrawals. On an announcement with an empty path the prefix handler
// deletes the peer's route, and the AS and peering handlers record nothing.
// UnmarshalJSON implements the json.Unmarshaler interface for ASPath.
func (p *ASPath) UnmarshalJSON(data []byte) error {
path, err := decodeASPath(data)
if err != nil {
return err
}
if slices.Contains(path.Sequence, 0) || slices.Contains(path.Set, 0) {
path = ASPath{}
}
*p = path
return nil
}
// decodeASPath decodes an AS path as the feed writes it.
func decodeASPath(data []byte) (ASPath, error) {
// Most paths have no AS set and decode as a plain array of integers.
var sequence []int
err := json.Unmarshal(data, &sequence)
if err == nil {
return ASPath{Sequence: sequence}, nil
*p = ASPath{Sequence: sequence}
return nil
}
var raw []any
err = json.Unmarshal(data, &raw)
if err != nil {
return ASPath{}, err
return err
}
path := ASPath{Sequence: make([]int, 0, len(raw))}
@@ -110,7 +90,9 @@ func decodeASPath(data []byte) (ASPath, error) {
}
}
return path, nil
*p = path
return nil
}
// RISLiveMessage represents the outer wrapper message from the RIPE RIS Live stream.
-30
View File
@@ -145,33 +145,3 @@ func TestASPathKeepsATrailingSetApart(t *testing.T) {
})
}
}
// TestASPathWithASZeroDecodesAsAnEmptyPath decodes AS paths that hold AS 0
// before their end, at their end, or in an AS set. RFC 7607 makes such a path
// malformed, so each decodes as an empty path.
func TestASPathWithASZeroDecodesAsAnEmptyPath(t *testing.T) {
t.Parallel()
for _, path := range []string{
"[64496, 0, 64497]",
"[64496, 0]",
"[64496, [0, 64497], 64498]",
"[64496, [64497, 0]]",
} {
t.Run(path, func(t *testing.T) {
t.Parallel()
var got ristypes.ASPath
err := json.Unmarshal([]byte(path), &got)
if err != nil {
t.Fatalf("unmarshal %s: %v", path, err)
}
if len(got.Sequence) > 0 || len(got.Set) > 0 {
t.Errorf("%s decoded to sequence %v and set %v, want an empty path",
path, got.Sequence, got.Set)
}
})
}
}
+5 -6
View File
@@ -116,18 +116,17 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
return err
}
// Start ASN WHOIS fetcher for background updates. The HTTP handlers read
// the server's fetcher without a lock, so it is set before they run.
rw.asnFetcher = NewASNFetcher(rw.db, rw.logger.Logger)
rw.asnFetcher.Start(ctx)
rw.server.SetASNFetcher(rw.asnFetcher)
// Start HTTP server
err = rw.server.Start(ctx)
if err != nil {
return err
}
// Start ASN WHOIS fetcher for background updates
rw.asnFetcher = NewASNFetcher(rw.db, rw.logger.Logger)
rw.asnFetcher.Start(ctx)
rw.server.SetASNFetcher(rw.asnFetcher)
// Start database maintenance goroutine
rw.dbMaintainer = NewDBMaintainer(rw.db, rw.logger.Logger)
rw.dbMaintainer.Start(ctx)
@@ -152,7 +152,7 @@ func (m *mockStore) RecordAnnouncement(_ *database.Announcement) error {
}
// RecordPeering mock implementation
func (m *mockStore) RecordPeering(asA, asB int, _ database.SeenTimes) error {
func (m *mockStore) RecordPeering(asA, asB int, _ time.Time) error {
m.mu.Lock()
defer m.mu.Unlock()
@@ -396,16 +396,16 @@ func (m *mockStore) DeleteLiveRoutesOfPeer(_ string) (int, error) {
}
// GetOrCreateASNBatch mock implementation
func (m *mockStore) GetOrCreateASNBatch(asns map[int]database.SeenTimes) error {
func (m *mockStore) GetOrCreateASNBatch(asns map[int]time.Time) error {
m.mu.Lock()
defer m.mu.Unlock()
for number, seen := range asns {
for number, timestamp := range asns {
if _, exists := m.ASNs[number]; !exists {
m.ASNs[number] = &database.ASN{
ASN: number,
FirstSeen: seen.First,
LastSeen: seen.Last,
FirstSeen: timestamp,
LastSeen: timestamp,
}
m.ASNCount++
}
@@ -671,9 +671,8 @@ func freePort(t *testing.T) int {
// TestShutdownReturnsByItsDeadlineWhileAWriteIsBlocked starts the HTTP server
// and has the AS handler start a write that the store holds until the test
// ends, past Shutdown's deadline. Shutdown must return within its deadline
// plus a small margin, with the HTTP server stopped, and must neither
// checkpoint nor close the database: the real one would wait for the write.
// Before the fix Shutdown waited for the write.
// plus a small margin, with the HTTP server stopped. Before the fix Shutdown
// waited for the write.
func TestShutdownReturnsByItsDeadlineWhileAWriteIsBlocked(t *testing.T) {
t.Parallel()
@@ -684,10 +683,9 @@ func TestShutdownReturnsByItsDeadlineWhileAWriteIsBlocked(t *testing.T) {
margin = 2 * time.Second
)
blocking := newBlockingStore()
defer blocking.releaseWrites()
store := newBlockingStore()
defer store.releaseWrites()
store := &closeOrderStore{Store: blocking}
log := logger.New()
port := freePort(t)
cfg := &config.Config{Port: port}
@@ -708,8 +706,7 @@ func TestShutdownReturnsByItsDeadlineWhileAWriteIsBlocked(t *testing.T) {
rw.dbHandler = NewASHandler(store, log)
fill := ristypes.ASPath{Sequence: slices.Repeat([]int{testASNA}, asnBatchSize)}
startBlockedWrite(t, blocking, rw.dbHandler,
[]*ristypes.RISMessage{{Path: fill}})
startBlockedWrite(t, store, rw.dbHandler, []*ristypes.RISMessage{{Path: fill}})
ctx, cancel := context.WithTimeout(t.Context(), deadline)
defer cancel()
@@ -739,21 +736,12 @@ func TestShutdownReturnsByItsDeadlineWhileAWriteIsBlocked(t *testing.T) {
t.Errorf("the HTTP server still accepts connections on %s after "+
"Shutdown returned", addr)
}
store.callsMu.Lock()
defer store.callsMu.Unlock()
if slices.Contains(store.calls, callCheckpoint) ||
slices.Contains(store.calls, callClose) {
t.Errorf("Shutdown made the calls %q past its deadline, want no %s "+
"and no %s", store.calls, callCheckpoint, callClose)
}
}
// closeOrderStore is a store that records, in order, the AS writes,
// checkpoints and closes it is given. It passes the AS writes on to Store.
// closeOrderStore is a mockStore that records, in order, the AS writes,
// checkpoints and closes it is given.
type closeOrderStore struct {
database.Store
*mockStore
callsMu sync.Mutex
calls []string
@@ -766,12 +754,10 @@ const (
callClose = "close"
)
func (s *closeOrderStore) GetOrCreateASNBatch(
asns map[int]database.SeenTimes,
) error {
func (s *closeOrderStore) GetOrCreateASNBatch(asns map[int]time.Time) error {
s.record(callWrite)
return s.Store.GetOrCreateASNBatch(asns)
return s.mockStore.GetOrCreateASNBatch(asns)
}
func (s *closeOrderStore) Checkpoint(_ context.Context) error {
@@ -802,7 +788,7 @@ func TestShutdownCheckpointsAndClosesTheDatabaseLast(t *testing.T) {
const deadline = 10 * time.Second
store := &closeOrderStore{Store: newMockStore()}
store := &closeOrderStore{mockStore: newMockStore()}
log := logger.New()
cfg := &config.Config{}
s := streamer.New(log, metrics.New())
+7 -9
View File
@@ -229,21 +229,19 @@ func (h *ASHandler) flushBatch() {
return
}
// The earliest and the latest time the batch saw each AS. The batch holds
// several updates for an AS, from many peers, so a new AS is stored first
// seen at the earliest of them and last seen at the latest.
asns := make(map[int]database.SeenTimes)
// Process ASNs first (deduped)
asnMap := make(map[int]time.Time)
for _, op := range batch {
seen := asns[op.number]
seen.Add(op.timestamp)
asns[op.number] = seen
if existing, ok := asnMap[op.number]; !ok || op.timestamp.After(existing) {
asnMap[op.number] = op.timestamp
}
}
// Process all ASNs in a single batch transaction
err := h.db.GetOrCreateASNBatch(asns)
err := h.db.GetOrCreateASNBatch(asnMap)
if err != nil {
h.logger.Error("Failed to process ASN batch",
"error", err, "count", len(asns))
"error", err, "count", len(asnMap))
}
h.mu.Lock()
@@ -1,68 +0,0 @@
package routewatch
import (
"maps"
"slices"
"testing"
"git.eeqj.de/sneak/routewatch/internal/logger"
)
// TestASHandlerStoresTheEarliestAndLatestTimesOfANewAS handles two updates
// whose path carries an AS the database does not hold yet, the second 5
// seconds after the first, in one batch into a real database. The AS is stored
// first seen at the time of the first update and last seen at the time of the
// second.
func TestASHandlerStoresTheEarliestAndLatestTimesOfANewAS(t *testing.T) {
t.Parallel()
db := newTestDatabase(t)
// Built without NewASHandler's flush goroutine, so the test flushes itself.
h := &ASHandler{db: db, logger: logger.New()}
first := peerAnnouncement(testPeerIP, testV4Prefix)
second := peerAnnouncement(testPeerIP, testV4Prefix)
second.ParsedTimestamp = first.ParsedTimestamp.Add(secondUpdateAfter)
h.HandleMessage(first)
h.HandleMessage(second)
h.flushBatch()
asn, _, err := db.GetASDetails(testASNA)
if err != nil {
t.Fatalf("GetASDetails(%d): %v", testASNA, err)
}
if !asn.FirstSeen.Equal(first.ParsedTimestamp) {
t.Errorf("first_seen of AS%d = %s, want %s", testASNA,
asn.FirstSeen, first.ParsedTimestamp)
}
if !asn.LastSeen.Equal(second.ParsedTimestamp) {
t.Errorf("last_seen of AS%d = %s, want %s", testASNA,
asn.LastSeen, second.ParsedTimestamp)
}
}
// TestASHandlerRecordsNothingFromAPathWithASZero handles an update whose AS
// path holds AS 0 between two other ASes. RFC 7607 makes such a path
// malformed, so neither AS is recorded.
func TestASHandlerRecordsNothingFromAPathWithASZero(t *testing.T) {
t.Parallel()
store := newMockStore()
// Built without NewASHandler's flush goroutine, so the test flushes itself.
h := &ASHandler{db: store, logger: logger.New()}
msg := peerAnnouncement(testPeerIP, testV4Prefix)
msg.Path = decodePath(t, testASNA, 0, testASNB)
h.HandleMessage(msg)
h.flushBatch()
if len(store.ASNs) > 0 {
t.Errorf("ASes recorded = %v, want none",
slices.Sorted(maps.Keys(store.ASNs)))
}
}
+13 -18
View File
@@ -53,8 +53,7 @@ type ASNFetcher struct {
currentInterval time.Duration
consecutiveFails int
// hourly stats tracking; now is the clock, time.Now except in tests
now func() time.Time
// hourly stats tracking
statsMu sync.Mutex
successTimes []time.Time
errorTimes []time.Time
@@ -70,7 +69,6 @@ func NewASNFetcher(db database.Store, logger *slog.Logger) *ASNFetcher {
stopCh: make(chan struct{}),
done: make(chan struct{}),
currentInterval: baseInterval,
now: time.Now,
successTimes: make([]time.Time, 0),
errorTimes: make([]time.Time, 0),
}
@@ -123,7 +121,7 @@ func (f *ASNFetcher) GetStats() server.ASNFetcherStats {
f.intervalMu.Unlock()
// Prune old entries and count
cutoff := f.now().Add(-statsWindow)
cutoff := time.Now().Add(-statsWindow)
f.successTimes = pruneOldTimes(f.successTimes, cutoff)
f.errorTimes = pruneOldTimes(f.errorTimes, cutoff)
@@ -135,15 +133,16 @@ func (f *ASNFetcher) GetStats() server.ASNFetcherStats {
}
}
// pruneOldTimes drops the times at or before cutoff from the front of times,
// which holds times in the order they were recorded, and returns the rest. It
// copies nothing, so it is cheap enough to run on every append.
// pruneOldTimes removes times older than cutoff and returns the pruned slice.
func pruneOldTimes(times []time.Time, cutoff time.Time) []time.Time {
for len(times) > 0 && !times[0].After(cutoff) {
times = times[1:]
result := make([]time.Time, 0, len(times))
for _, t := range times {
if t.After(cutoff) {
result = append(result, t)
}
}
return times
return result
}
// getInterval returns the current fetch interval.
@@ -171,11 +170,9 @@ func (f *ASNFetcher) recordSuccess() {
}
f.intervalMu.Unlock()
// Record success time for stats, dropping those older than statsWindow
// Record success time for stats
f.statsMu.Lock()
now := f.now()
f.successTimes = pruneOldTimes(f.successTimes, now.Add(-statsWindow))
f.successTimes = append(f.successTimes, now)
f.successTimes = append(f.successTimes, time.Now())
f.statsMu.Unlock()
}
@@ -197,11 +194,9 @@ func (f *ASNFetcher) recordFailure() {
}
f.intervalMu.Unlock()
// Record error time for stats, dropping those older than statsWindow
// Record error time for stats
f.statsMu.Lock()
now := f.now()
f.errorTimes = pruneOldTimes(f.errorTimes, now.Add(-statsWindow))
f.errorTimes = append(f.errorTimes, now)
f.errorTimes = append(f.errorTimes, time.Now())
f.statsMu.Unlock()
}
@@ -155,83 +155,3 @@ func TestFetchAndUpdateMarksAnASWithoutReferral(t *testing.T) {
baseInterval)
}
}
// TestASNFetcherStopReturnsByItsDeadline starts a fetcher, has it look up an
// AS whose WHOIS server does not answer until the test ends, and stops it with
// a short deadline while that lookup is in progress. Stop must return all the
// same instead of waiting for the lookup.
func TestASNFetcherStopReturnsByItsDeadline(t *testing.T) {
t.Parallel()
const stopDeadline = 100 * time.Millisecond
lookupStarted := make(chan struct{})
release := make(chan struct{})
f := NewASNFetcher(newMockStore(), logger.New().Logger)
f.whoisClient.DialContext = func(
context.Context, string, string,
) (net.Conn, error) {
close(lookupStarted)
<-release
return nil, syscall.ECONNREFUSED
}
f.Start(t.Context())
f.QueueImmediate(testASNA)
select {
case <-lookupStarted:
case <-time.After(handleMessageTimeout):
t.Fatal("the fetcher did not start the lookup")
}
ctx, cancel := context.WithTimeout(t.Context(), stopDeadline)
defer cancel()
if !returnsInTime(func() { f.Stop(ctx) }) {
t.Fatalf("Stop with a deadline of %s did not return while a lookup "+
"was in progress", stopDeadline)
}
close(release)
select {
case <-f.done:
case <-time.After(handleMessageTimeout):
t.Fatal("the fetcher goroutine did not return after the lookup ended")
}
}
// TestRecordKeepsOnlyTheStatsWindow records a success and a failure every
// second, by a clock the test sets, far longer than statsWindow, and never
// asks for the stats. The fetcher must keep only the times within the last
// statsWindow.
func TestRecordKeepsOnlyTheStatsWindow(t *testing.T) {
t.Parallel()
const (
records = 100_000
step = time.Second
want = int(statsWindow / step)
)
clock := time.Now()
f := NewASNFetcher(newMockStore(), logger.New().Logger)
f.now = func() time.Time { return clock }
for range records {
clock = clock.Add(step)
f.recordSuccess()
f.recordFailure()
}
if len(f.successTimes) != want || len(f.errorTimes) != want {
t.Errorf("%d success and %d error times kept, want %d of each, "+
"those of the last %s", len(f.successTimes), len(f.errorTimes),
want, statsWindow)
}
}
@@ -66,7 +66,7 @@ func (s *blockingStore) UpsertLiveRouteBatch(routes []*database.LiveRoute) error
return nil
}
func (s *blockingStore) GetOrCreateASNBatch(asns map[int]database.SeenTimes) error {
func (s *blockingStore) GetOrCreateASNBatch(asns map[int]time.Time) error {
s.waitForRelease()
s.mu.Lock()
defer s.mu.Unlock()
@@ -96,12 +96,10 @@ func (s *blockingStore) DeleteLiveRoutesOfPeer(peerIP string) (int, error) {
return s.mockStore.DeleteLiveRoutesOfPeer(peerIP)
}
func (s *blockingStore) RecordPeering(
asA, asB int, seen database.SeenTimes,
) error {
func (s *blockingStore) RecordPeering(asA, asB int, ts time.Time) error {
s.waitForRelease()
return s.mockStore.RecordPeering(asA, asB, seen)
return s.mockStore.RecordPeering(asA, asB, ts)
}
func (s *blockingStore) waitForRelease() {
+20 -4
View File
@@ -3,8 +3,10 @@ package routewatch
import (
"context"
"os"
"os/signal"
"runtime"
"strings"
"syscall"
"time"
"git.eeqj.de/sneak/routewatch/internal/logger"
@@ -60,10 +62,10 @@ func logDebugStats(logger *logger.Logger) {
}
// CLIEntry is the main entry point for the routewatch command-line interface.
// It initializes the application using the fx dependency injection framework
// and starts the RouteWatch service. It blocks until fx receives SIGINT or
// SIGTERM or RouteWatch stops on an error; fx then runs the stop hook and sets
// the exit code.
// It initializes the application using the fx dependency injection framework,
// sets up signal handling for graceful shutdown, and starts the RouteWatch
// service. This function blocks until the application receives a shutdown
// signal or encounters a fatal error.
func CLIEntry() {
app := fx.New(
fx.Options(getModule()...),
@@ -79,6 +81,20 @@ func CLIEntry() {
// Start debug stats logging
go logDebugStats(logger)
// Handle shutdown signals
sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, os.Interrupt, syscall.SIGTERM)
go func() {
<-sigCh
logger.Info("Received shutdown signal")
err := shutdowner.Shutdown()
if err != nil {
logger.Error("Failed to shutdown gracefully", "error", err)
}
}()
go func() {
err := rw.Run(ctx)
if err != nil {
@@ -6,7 +6,6 @@ import (
"encoding/json"
"log/slog"
"testing"
"time"
"git.eeqj.de/sneak/routewatch/internal/database"
)
@@ -53,68 +52,3 @@ func TestIncompleteCheckpointNotLoggedAsCompleted(t *testing.T) {
m.checkpointCount)
}
}
// heldCheckpointStore is a store whose checkpoint closes started and then
// waits until release is closed.
type heldCheckpointStore struct {
*mockStore
started chan struct{}
release chan struct{}
}
// Checkpoint waits until release is closed.
func (s heldCheckpointStore) Checkpoint(context.Context) error {
close(s.started)
<-s.release
return nil
}
// TestDBMaintainerStopReturnsByItsDeadline stops the maintainer with a short
// deadline while its goroutine runs a checkpoint that the store holds until
// the test ends. Stop must return all the same instead of waiting for the
// checkpoint.
func TestDBMaintainerStopReturnsByItsDeadline(t *testing.T) {
t.Parallel()
const stopDeadline = 100 * time.Millisecond
store := heldCheckpointStore{
mockStore: newMockStore(),
started: make(chan struct{}),
release: make(chan struct{}),
}
m := NewDBMaintainer(store, slog.Default())
// Start would run the first checkpoint only after checkpointInterval.
// This goroutine stands for the one Start runs and checkpoints at once.
go func() {
defer close(m.done)
m.runCheckpoint(t.Context())
}()
select {
case <-store.started:
case <-time.After(handleMessageTimeout):
t.Fatal("the maintainer did not start the checkpoint")
}
ctx, cancel := context.WithTimeout(t.Context(), stopDeadline)
defer cancel()
if !returnsInTime(func() { m.Stop(ctx) }) {
t.Fatalf("Stop with a deadline of %s did not return while a "+
"checkpoint was in progress", stopDeadline)
}
close(store.release)
select {
case <-m.done:
case <-time.After(handleMessageTimeout):
t.Fatal("the maintenance goroutine did not return after the " +
"checkpoint ended")
}
}
+19 -16
View File
@@ -37,8 +37,8 @@ const (
// PeerHandler tracks BGP peers from UPDATE messages using batched operations.
// It maintains a queue of peer updates and periodically flushes them to the database
// in batches to improve performance. The handler writes one row per peer IP
// address for each batch, with the earliest and the latest time of its updates.
// in batches to improve performance. The handler deduplicates peer updates within
// each batch, keeping only the most recent update for each peer IP address.
type PeerHandler struct {
db database.Store
logger *logger.Logger
@@ -227,28 +227,31 @@ func (h *PeerHandler) flushBatch() {
return
}
// The earliest and the latest time the batch saw each peer, with the peer
// AS and message type of its latest update. The batch holds several
// updates for a peer, so a new peer is stored first seen at the earliest of
// them and last seen at the latest.
peers := make(map[string]database.PeerUpdate)
// Deduplicate by peer IP, keeping the latest update for each peer
peerMap := make(map[string]peerUpdate)
for _, update := range batch {
peer, ok := peers[update.peerIP]
if !ok || update.timestamp.After(peer.Seen.Last) {
peer.PeerIP = update.peerIP
peer.PeerASN = update.peerASN
peer.MessageType = update.messageType
existing, ok := peerMap[update.peerIP]
if !ok || update.timestamp.After(existing.timestamp) {
peerMap[update.peerIP] = update
}
}
peer.Seen.Add(update.timestamp)
peers[update.peerIP] = peer
// Convert to database format
dbPeerMap := make(map[string]database.PeerUpdate)
for peerIP, update := range peerMap {
dbPeerMap[peerIP] = database.PeerUpdate{
PeerIP: update.peerIP,
PeerASN: update.peerASN,
MessageType: update.messageType,
Timestamp: update.timestamp,
}
}
// Process all peers in a single batch transaction
err := h.db.UpdatePeerBatch(peers)
err := h.db.UpdatePeerBatch(dbPeerMap)
if err != nil {
h.logger.Error("Failed to process peer batch",
"error", err, "count", len(peers))
"error", err, "count", len(dbPeerMap))
}
h.mu.Lock()
@@ -1,59 +0,0 @@
package routewatch
import (
"testing"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
)
// peerBatchStore keeps the peers of the last UpdatePeerBatch call. The
// database has no read of the peer table, so the peer handler's test checks
// what the handler writes.
type peerBatchStore struct {
*mockStore
peers map[string]database.PeerUpdate
}
func (s *peerBatchStore) UpdatePeerBatch(
peers map[string]database.PeerUpdate,
) error {
s.peers = peers
return nil
}
// TestPeerHandlerWritesTheEarliestAndLatestTimesOfANewPeer handles two updates
// from one peer, the second 5 seconds after the first, in one batch. The peer
// handler writes the peer first seen at the time of the first update and last
// seen at the time of the second.
func TestPeerHandlerWritesTheEarliestAndLatestTimesOfANewPeer(t *testing.T) {
t.Parallel()
store := &peerBatchStore{mockStore: newMockStore()}
// Built without NewPeerHandler's flush goroutine, so the test flushes
// itself.
h := &PeerHandler{db: store, logger: logger.New()}
first := peerAnnouncement(testPeerIP, testV4Prefix)
second := peerAnnouncement(testPeerIP, testV4Prefix)
second.ParsedTimestamp = first.ParsedTimestamp.Add(secondUpdateAfter)
h.HandleMessage(first)
h.HandleMessage(second)
h.flushBatch()
seen := store.peers[testPeerIP].Seen
if !seen.First.Equal(first.ParsedTimestamp) {
t.Errorf("first_seen of peer %s = %s, want %s", testPeerIP,
seen.First, first.ParsedTimestamp)
}
if !seen.Last.Equal(second.ParsedTimestamp) {
t.Errorf("last_seen of peer %s = %s, want %s", testPeerIP,
seen.Last, second.ParsedTimestamp)
}
}
+21 -31
View File
@@ -42,11 +42,10 @@ type PeeringHandler struct {
db database.Store
logger *logger.Logger
// In-memory AS path tracking: each path, JSON-encoded, with the earliest
// and the latest time an update carried it.
// In-memory AS path tracking
mu sync.Mutex
asPaths map[string]database.SeenTimes
droppedPaths int // paths dropped since the last run because the map was full
asPaths map[string]time.Time // key is JSON-encoded AS path
droppedPaths int // paths dropped because the map was full
// writing is how many paths processPeerings took from asPaths and has
// not finished writing the peerings of.
writing int
@@ -65,7 +64,7 @@ func NewPeeringHandler(db database.Store, logger *logger.Logger) *PeeringHandler
h := &PeeringHandler{
db: db,
logger: logger,
asPaths: make(map[string]database.SeenTimes),
asPaths: make(map[string]time.Time),
stopCh: make(chan context.Context, 1),
done: make(chan struct{}),
}
@@ -119,18 +118,16 @@ func (h *PeeringHandler) HandleMessage(msg *ristypes.RISMessage) {
key := string(pathJSON)
h.mu.Lock()
defer h.mu.Unlock()
seen, exists := h.asPaths[key]
if !exists && len(h.asPaths) >= maxTrackedPaths {
if _, exists := h.asPaths[key]; exists {
// Already tracked: refresh its timestamp.
h.asPaths[key] = timestamp
} else if len(h.asPaths) >= maxTrackedPaths {
// Map is full; drop this new path and count it.
h.droppedPaths++
return
} else {
h.asPaths[key] = timestamp
}
seen.Add(timestamp)
h.asPaths[key] = seen
h.mu.Unlock()
}
// ProcessPeeringsNow triggers immediate processing of all accumulated AS
@@ -190,14 +187,12 @@ func (h *PeeringHandler) processLoop() {
func (h *PeeringHandler) processPeerings() {
// Take the accumulated paths and replace the map with a fresh empty one
// under the lock. Each path is processed exactly once and the memory is
// released, so the map never grows past a single interval's traffic. The
// count of dropped paths starts again with the new map.
// released, so the map never grows past a single interval's traffic.
h.mu.Lock()
paths := h.asPaths
h.asPaths = make(map[string]database.SeenTimes)
h.asPaths = make(map[string]time.Time)
h.writing = len(paths)
dropped := h.droppedPaths
h.droppedPaths = 0
h.mu.Unlock()
if len(paths) == 0 {
@@ -209,13 +204,9 @@ func (h *PeeringHandler) processPeerings() {
low, high int
}
// The earliest and the latest time an update carried each peering. Several
// paths, each carried at several times, can hold a peering, so a new
// peering is stored first seen at the earliest of them and last seen at the
// latest.
peerings := make(map[peeringKey]database.SeenTimes)
peerings := make(map[peeringKey]time.Time)
for pathJSON, pathSeen := range paths {
for pathJSON, timestamp := range paths {
var path []int
err := json.Unmarshal([]byte(pathJSON), &path)
@@ -242,11 +233,10 @@ func (h *PeeringHandler) processPeerings() {
}
key := peeringKey{low: low, high: high}
seen := peerings[key]
seen.Add(pathSeen.First)
seen.Add(pathSeen.Last)
peerings[key] = seen
// Update timestamp if this is newer
if existing, ok := peerings[key]; !ok || timestamp.After(existing) {
peerings[key] = timestamp
}
}
}
@@ -254,8 +244,8 @@ func (h *PeeringHandler) processPeerings() {
start := time.Now()
successCount := 0
for key, seen := range peerings {
err := h.db.RecordPeering(key.low, key.high, seen)
for key, ts := range peerings {
err := h.db.RecordPeering(key.low, key.high, ts)
if err != nil {
h.logger.Error("Failed to record peering",
"as_a", key.low,
@@ -31,14 +31,12 @@ type recordingStore struct {
calls int
}
func (r *recordingStore) RecordPeering(
asA, asB int, seen database.SeenTimes,
) error {
func (r *recordingStore) RecordPeering(asA, asB int, ts time.Time) error {
r.mu.Lock()
r.calls++
r.mu.Unlock()
return r.mockStore.RecordPeering(asA, asB, seen)
return r.mockStore.RecordPeering(asA, asB, ts)
}
func (r *recordingStore) callCount() int {
@@ -54,7 +52,7 @@ func newTestHandler(db database.Store) *PeeringHandler {
return &PeeringHandler{
db: db,
logger: logger.New(),
asPaths: make(map[string]database.SeenTimes),
asPaths: make(map[string]time.Time),
}
}
@@ -153,43 +151,6 @@ func TestPeeringHandlerLeavesATrailingASSetOutOfThePeerings(t *testing.T) {
}
}
// TestPeeringHandlerRecordsNothingFromAPathWithASZero handles AS paths that
// hold AS 0. RFC 7607 makes such a path malformed, so no peering is recorded
// from it, not even one between two other ASes of the path.
func TestPeeringHandlerRecordsNothingFromAPathWithASZero(t *testing.T) {
t.Parallel()
tests := []struct {
name string
path []any
}{
{name: "AS 0 between two ASes", path: []any{testASNA, 0, testASNB}},
{
name: "AS 0 after two ASes",
path: []any{testASNA, testASNB, 0, testASNC},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
store := newMockStore()
h := newTestHandler(store)
h.HandleMessage(&ristypes.RISMessage{
Path: decodePath(t, tt.path...),
ParsedTimestamp: time.Now().UTC(),
})
h.ProcessPeeringsNow()
if len(store.Peerings) > 0 {
t.Errorf("peerings recorded = %v, want none",
slices.Sorted(maps.Keys(store.Peerings)))
}
})
}
}
// TestPeeringHandlerProcessesEachRunAndEmpties verifies that a processing run
// empties the path map (the swap) and that a path seen again after a run is
// recorded in the next run too.
@@ -259,10 +220,10 @@ func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
existing := pathKey(t, testASNA, testASNB)
h.mu.Lock()
h.asPaths[existing] = database.SeenTimes{First: now, Last: now}
h.asPaths[existing] = now
for i := 0; len(h.asPaths) < maxTrackedPaths; i++ {
h.asPaths[strconv.Itoa(i)] = database.SeenTimes{First: now, Last: now}
h.asPaths[strconv.Itoa(i)] = now
}
h.mu.Unlock()
@@ -292,83 +253,11 @@ func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
}
h.mu.Lock()
gotSeen := h.asPaths[existing]
gotTS := h.asPaths[existing]
h.mu.Unlock()
if !gotSeen.Last.Equal(refreshed) {
t.Fatalf("existing path last seen not refreshed: want %v, got %v",
refreshed, gotSeen.Last)
}
}
// TestPeeringHandlerRunResetsTheDroppedCount fills the map, has the handler
// drop one path, and runs the processing. The count of dropped paths is the
// run's own, so it is 0 again after the run.
func TestPeeringHandlerRunResetsTheDroppedCount(t *testing.T) {
t.Parallel()
h := newTestHandler(newMockStore())
now := time.Now().UTC()
// Paths of one AS carry no peering, so the run records nothing.
h.mu.Lock()
for asn := 1; len(h.asPaths) < maxTrackedPaths; asn++ {
h.asPaths[pathKey(t, asn)] = database.SeenTimes{First: now, Last: now}
}
h.mu.Unlock()
handle(h, now, testASNA, testASNB)
if _, dropped := snapshot(h); dropped != 1 {
t.Fatalf("want dropped count 1 with the map full, got %d", dropped)
}
h.ProcessPeeringsNow()
if _, dropped := snapshot(h); dropped != 0 {
t.Errorf("want dropped count 0 after a run, got %d", dropped)
}
}
// TestPeeringHandlerStoresTheEarliestAndLatestTimesOfANewPeering handles two
// updates whose path carries a peering the database does not hold yet, the
// second 5 seconds after the first, in one run into a real database. The
// peering is stored first seen at the time of the first update and last seen
// at the time of the second.
func TestPeeringHandlerStoresTheEarliestAndLatestTimesOfANewPeering(
t *testing.T,
) {
t.Parallel()
db := newTestDatabase(t)
h := newTestHandler(db)
first := peerAnnouncement(testPeerIP, testV4Prefix)
second := peerAnnouncement(testPeerIP, testV4Prefix)
second.ParsedTimestamp = first.ParsedTimestamp.Add(secondUpdateAfter)
h.HandleMessage(first)
h.HandleMessage(second)
h.ProcessPeeringsNow()
peers, err := db.GetASPeers(testASNA)
if err != nil {
t.Fatalf("GetASPeers(%d): %v", testASNA, err)
}
if len(peers) != 1 || peers[0].ASN != testASNB {
t.Fatalf("peers of AS%d = %+v, want AS%d alone", testASNA, peers,
testASNB)
}
if !peers[0].FirstSeen.Equal(first.ParsedTimestamp) {
t.Errorf("first_seen of peering AS%d-AS%d = %s, want %s", testASNA,
testASNB, peers[0].FirstSeen, first.ParsedTimestamp)
}
if !peers[0].LastSeen.Equal(second.ParsedTimestamp) {
t.Errorf("last_seen of peering AS%d-AS%d = %s, want %s", testASNA,
testASNB, peers[0].LastSeen, second.ParsedTimestamp)
if !gotTS.Equal(refreshed) {
t.Fatalf("existing path timestamp not refreshed: want %v, got %v",
refreshed, gotTS)
}
}
+14 -21
View File
@@ -44,9 +44,8 @@ const (
)
// PrefixHandler tracks BGP prefixes and maintains a live routing table in the
// database. Routes are added on announcement and deleted on withdrawal or on an
// announcement whose AS path is empty or holds AS 0 anywhere, and all routes of
// a peer are deleted when its BGP session goes down.
// database. Routes are added on announcement and deleted on withdrawal, and all
// routes of a peer are deleted when its BGP session goes down.
type PrefixHandler struct {
db database.Store
logger *logger.Logger
@@ -348,19 +347,8 @@ func (h *PrefixHandler) flushBatch() {
lastUpdates := make(map[prefixAndPeer]prefixUpdate)
// Each prefix the batch names, announced or withdrawn, with the earliest
// timestamp among all of its updates, not only each peer's last one: the
// prefix tables store only first_seen, written when the prefix is new and
// never changed after.
prefixesToUpdate := make(map[string]time.Time)
for _, update := range batch {
lastUpdates[prefixAndPeer{update.prefix, update.peer}] = update
seen, exists := prefixesToUpdate[update.prefix]
if !exists || update.timestamp.Before(seen) {
prefixesToUpdate[update.prefix] = update.timestamp
}
}
// Collect routes to upsert and delete
@@ -369,14 +357,19 @@ func (h *PrefixHandler) flushBatch() {
routesToDelete []database.LiveRouteDeletion
)
// Collect unique prefixes to update
prefixesToUpdate := make(map[string]time.Time)
for _, update := range lastUpdates {
// Track prefix for both announcements and withdrawals
seen, exists := prefixesToUpdate[update.prefix]
if !exists || update.timestamp.After(seen) {
prefixesToUpdate[update.prefix] = update.timestamp
}
// An announcement is stored when its path gives an origin, or when it
// ends in an AS set of several members: that route is stored with
// origin 0 (none). Any other update deletes the peer's route for the
// prefix: a withdrawal, or an announcement whose path is empty or
// holds AS 0 anywhere (the decoder empties such a path). Such an
// announcement replaces the peer's route, as any announcement does,
// with none (RFC 7607 treats AS 0 in a path as a withdrawal).
// origin 0 (none). An announcement with an empty path is not stored.
if update.messageType == "announcement" &&
(update.originASN > 0 || update.endsInSet) {
// Create live route for batch upsert
@@ -384,11 +377,11 @@ func (h *PrefixHandler) flushBatch() {
if route != nil {
routesToUpsert = append(routesToUpsert, route)
}
} else {
} else if update.messageType == "withdrawal" {
// Parse CIDR to get IP version
_, ipVersion, err := parseCIDR(update.prefix)
if err != nil {
h.logger.Error("Failed to parse CIDR for route deletion",
h.logger.Error("Failed to parse CIDR for withdrawal",
"prefix", update.prefix, "error", err)
continue
@@ -26,10 +26,6 @@ const (
testV6Prefix = "2001:db8:1::/48"
)
// secondUpdateAfter is how long after the first update the tests of first_seen
// and last_seen send the second, in the same batch.
const secondUpdateAfter = 5 * time.Second
// TestPrefixHandlerStoresPrefixesTheIPLookupFinds runs announcements and
// withdrawals through the prefix handler into a real database and looks the
// addresses up. The prefixes are written the way the feed sends them: IPv6
@@ -87,49 +83,6 @@ func TestPrefixHandlerStoresPrefixesTheIPLookupFinds(t *testing.T) {
}
}
// TestPrefixHandlerStoresANewPrefixFirstSeenAtItsEarliestUpdate announces a
// prefix the database does not hold yet twice in one batch, the second time 5
// seconds after the first, through the prefix handler into a real database. The
// prefix is stored first seen at the time of the first announcement, whether
// the two come from two peers or from one, and the IP lookup answers with that
// time.
func TestPrefixHandlerStoresANewPrefixFirstSeenAtItsEarliestUpdate(
t *testing.T,
) {
t.Parallel()
const otherPeerIP = "2001:db8:ffff::2"
tests := []struct {
name string
firstPeer, secondPeer string
}{
{name: "two peers", firstPeer: testPeerIP, secondPeer: otherPeerIP},
{name: "one peer", firstPeer: testPeerIP, secondPeer: testPeerIP},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
first := peerAnnouncement(tt.firstPeer, testV4Prefix)
second := peerAnnouncement(tt.secondPeer, testV4Prefix)
second.ParsedTimestamp = first.ParsedTimestamp.Add(secondUpdateAfter)
db := handleBatches(t, [][]*ristypes.RISMessage{{first, second}})
info, err := db.GetIPInfo("192.0.2.1")
if err != nil {
t.Fatalf("GetIPInfo(192.0.2.1): %v", err)
}
if !info.FirstSeen.Equal(first.ParsedTimestamp) {
t.Errorf("first_seen of %s = %s, want %s", testV4Prefix,
info.FirstSeen, first.ParsedTimestamp)
}
})
}
}
// TestPrefixHandlerAppliesEachPeersLastUpdateInABatch runs updates for one
// prefix through the prefix handler into a real database. All updates carry the
// same timestamp, as updates within one second of the feed do. Within a batch,
@@ -378,133 +331,6 @@ func TestPrefixHandlerDoesNotStoreAnAnnouncementWithAnEmptyPath(t *testing.T) {
assertPeers(t, db, map[string][]string{testV4Prefix: nil})
}
// TestPrefixHandlerRemovesTheRouteOnAnAnnouncementWithoutAnOrigin runs, through
// the prefix handler into a real database, a peer's announcement of a prefix
// with origin AS testASNB and then the peer's announcement of the prefix with
// an AS path that gives no origin: one whose last AS is 0, an empty one, or one
// that holds AS 0 before its end, which decodes as an empty path. As any later
// announcement from the peer does, the second replaces the peer's route, here
// with none, whether it comes in a later batch or follows a withdrawal in one
// batch. The in-memory route count and prefix distribution must match the
// route table.
func TestPrefixHandlerRemovesTheRouteOnAnAnnouncementWithoutAnOrigin(
t *testing.T,
) {
t.Parallel()
const prefixMaskLength = 24
noOrigin := func(path ristypes.ASPath) *ristypes.RISMessage {
msg := peerAnnouncement(testPeerIP, testV4Prefix)
msg.Path = path
return msg
}
endsInZero := ristypes.ASPath{Sequence: []int{testASNA, 0}}
withdrawal := &ristypes.RISMessage{
Type: ristypes.MessageTypeUpdate,
Peer: testPeerIP,
ParsedTimestamp: time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC),
Withdrawals: []string{testV4Prefix},
}
tests := []struct {
name string
// Handled and flushed as one batch after the announcement with origin
// AS testASNB.
batch []*ristypes.RISMessage
}{
{
name: "a path whose last AS is 0",
batch: []*ristypes.RISMessage{noOrigin(endsInZero)},
},
{
name: "an empty path",
batch: []*ristypes.RISMessage{noOrigin(ristypes.ASPath{})},
},
{
name: "a path with AS 0 before its end",
batch: []*ristypes.RISMessage{
noOrigin(decodePath(t, testASNA, 0, testASNB)),
},
},
{
name: "a withdrawal and then a path whose last AS is 0",
batch: []*ristypes.RISMessage{withdrawal, noOrigin(endsInZero)},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
db := handleBatches(t, [][]*ristypes.RISMessage{
{peerAnnouncement(testPeerIP, testV4Prefix)},
tt.batch,
})
checkLivePeers(t, db, testV4Prefix, prefixMaskLength, nil)
})
}
}
// TestPrefixHandlerDoesNotStoreAPrefixThatDoesNotParse runs an announcement of
// "not-a-prefix" and a withdrawal of "also/bad" through the prefix handler into
// a real database. Neither is stored in the prefix tables: the prefix count in
// the stats stays 0, both as kept in memory and as counted from the tables when
// the database is opened again.
func TestPrefixHandlerDoesNotStoreAPrefixThatDoesNotParse(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()}
db, err := database.New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
// Built without NewPrefixHandler's flush timer, so the test flushes itself.
h := &PrefixHandler{db: db, logger: logger.New()}
withdrawal := &ristypes.RISMessage{
Type: ristypes.MessageTypeUpdate,
Peer: testPeerIP,
ParsedTimestamp: time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC),
Withdrawals: []string{"also/bad"},
}
h.HandleMessage(peerAnnouncement(testPeerIP, "not-a-prefix"))
h.HandleMessage(withdrawal)
h.flushBatch()
checkNoPrefixes := func(when string) {
t.Helper()
stats, err := db.GetStatsContext(context.Background())
if err != nil {
t.Fatalf("GetStatsContext %s: %v", when, err)
}
if stats.Prefixes != 0 {
t.Errorf("prefix count %s = %d, want 0", when, stats.Prefixes)
}
}
checkNoPrefixes("after the flush")
err = db.Close()
if err != nil {
t.Fatalf("Close: %v", err)
}
db, err = database.New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to open the database again: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
checkNoPrefixes("after opening the database again")
}
// TestStateMessagesGoToThePrefixHandlerNotThePeerHandler checks that STATE
// messages go to the prefix handler, which acts on a down, and not to the peer
// handler, which would record a peer whose session never comes up as active.
@@ -970,25 +796,6 @@ func TestPrefixHandlerStopLogsAFailedDeleteWithoutTheDeadline(t *testing.T) {
}
}
// TestPrefixHandlerStopLogsNothingAfterADeleteThatSucceeds stops a prefix
// handler, with no deadline, after a peer's session went down, into a database
// whose delete of the peer's routes succeeds. Stop must log nothing: a peer
// whose routes are deleted no longer counts as a peer down left undone.
func TestPrefixHandlerStopLogsNothingAfterADeleteThatSucceeds(t *testing.T) {
t.Parallel()
var logs bytes.Buffer
h := NewPrefixHandler(newTestDatabase(t), newErrorLogger(&logs))
h.HandleMessage(sessionDown())
h.Stop(t.Context())
if logs.Len() != 0 {
t.Errorf("Stop logged %q after the routes of the peer whose session "+
"went down were deleted", logs.String())
}
}
// newTestDatabase opens a database in a new temporary directory. It is closed
// when the test ends.
func newTestDatabase(t *testing.T) *database.Database {
+48 -138
View File
@@ -31,18 +31,6 @@ const (
healthCheckTimeout = 2 * time.Second
)
// Limits of one IP lookup.
const (
// maxLookupTargets is the most addresses and host names an IP lookup may
// list, and the most addresses it may have once the names are resolved:
// each address costs up to 113 database reads and a PTR lookup.
maxLookupTargets = 10
// ptrLookupTimeout is each PTR lookup's own time limit, so that ten
// addresses whose PTR lookups never answer fit in the 30-second limit.
ptrLookupTimeout = 2 * time.Second
)
// Keys of the JSON envelope the handlers and middleware write, and the status
// value of an error answer.
const (
@@ -53,13 +41,10 @@ const (
jsonStatusError = "error"
)
// Error messages an answer sends, in place of the error behind it where there
// is one.
// Error messages an answer sends in place of the error that caused it.
const (
invalidPrefixMessage = "Invalid prefix"
hostNotResolvedMessage = "could not resolve host name"
hostNotPublicMessage = "host name has no public address"
tooManyTargetsMessage = "Too many addresses or host names"
)
// IP versions as database.LiveRoute.IPVersion records them.
@@ -690,18 +675,15 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
}
ctx := r.Context()
response := IPLookupResponse{Query: target, Results: make([]*database.IPInfo, 0)}
// Collect all IPs to look up
ipsToLookup, resolveErrors, err := s.resolveLookupTargets(ctx, target)
if deadlinePassed(ctx) {
s.writeJSONError(w, http.StatusGatewayTimeout, requestTimeoutMessage)
return
response := IPLookupResponse{
Query: target,
Results: make([]*database.IPInfo, 0),
}
if err != nil {
s.writeJSONError(w, http.StatusBadRequest, tooManyTargetsMessage)
// Collect all IPs to look up
ipsToLookup, resolveErrors := s.resolveLookupTargets(ctx, target)
if deadlinePassed(ctx) {
s.writeJSONError(w, http.StatusGatewayTimeout, requestTimeoutMessage)
return
}
@@ -709,7 +691,7 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
response.Errors = append(response.Errors, resolveErrors...)
if len(ipsToLookup) == 0 {
s.writeJSONError(w, http.StatusBadRequest, noTargetsMessage(resolveErrors))
s.writeJSONError(w, http.StatusBadRequest, "No valid IPs or hostnames provided")
return
}
@@ -726,20 +708,15 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
return
}
switch {
case errors.Is(err, database.ErrNoRoute), errors.Is(err, database.ErrInvalidIP):
response.Errors = append(response.Errors, ip+": "+err.Error())
if err != nil {
response.Errors = append(response.Errors,
ip+": "+s.lookupErrorMessage(ip, err))
continue
case err != nil:
s.logger.Error("Failed to look up IP", "ip", ip, "error", err)
s.writeJSONError(w, http.StatusInternalServerError, internalErrorMessage)
return
}
// Do PTR lookup for this IP
ptrs := s.lookupPTRNames(ctx, ip)
ptrs := lookupPTRNames(ctx, ip)
if deadlinePassed(ctx) {
s.writeJSONError(w, http.StatusGatewayTimeout, requestTimeoutMessage)
@@ -760,38 +737,31 @@ func (s *Server) handleIPInfo() http.HandlerFunc {
// Queue WHOIS refresh for stale ASNs (non-blocking)
s.queueWHOISRefresh(refreshASNs)
s.writeIPLookupAnswer(w, response)
// Return response (even if no results, include errors)
if len(response.Results) == 0 && len(response.Errors) > 0 {
s.writeJSONError(w, http.StatusNotFound, "No routes found: "+response.Errors[0])
return
}
err := writeJSONSuccess(w, response)
if err != nil {
s.logger.Error("Failed to encode IP lookup response", "error", err)
}
}
}
// noTargetsMessage returns the message of the 400 an IP lookup answers when
// no address is left to look up, followed by the first of resolveErrors, which
// says why, when there is one.
func noTargetsMessage(resolveErrors []string) string {
const message = "No valid IPs or hostnames provided"
if len(resolveErrors) == 0 {
return message
// lookupErrorMessage returns what an IP lookup answer says about err, the
// failed lookup of ip: err's own text when ip is not a valid address or has no
// route, otherwise a fixed message, with err in the log.
func (s *Server) lookupErrorMessage(ip string, err error) string {
if errors.Is(err, database.ErrNoRoute) || errors.Is(err, database.ErrInvalidIP) {
return err.Error()
}
return message + ": " + resolveErrors[0]
}
s.logger.Error("Failed to look up IP", "ip", ip, "error", err)
// writeIPLookupAnswer sends response, or 404 with its first error when no
// address had a route.
func (s *Server) writeIPLookupAnswer(
w http.ResponseWriter, response IPLookupResponse,
) {
if len(response.Results) == 0 && len(response.Errors) > 0 {
s.writeJSONError(w, http.StatusNotFound, "No routes found: "+response.Errors[0])
return
}
err := writeJSONSuccess(w, response)
if err != nil {
s.logger.Error("Failed to encode IP lookup response", "error", err)
}
return internalErrorMessage
}
// lookupTarget returns what an IP lookup request asks for: the ip or addr URL
@@ -822,16 +792,15 @@ func (s *Server) queueWHOISRefresh(asns map[int]bool) {
}
}
// errTooManyTargets is what resolveLookupTargets returns for a list of more
// than maxLookupTargets addresses and host names, or of names that resolve to
// more than maxLookupTargets addresses.
var errTooManyTargets = errors.New("too many addresses or host names")
// splitLookupTargets returns the addresses and host names in a comma-separated
// list, leaving out blank entries, or errTooManyTargets when there are more
// than maxLookupTargets.
func splitLookupTargets(target string) ([]string, error) {
var targets []string
// resolveLookupTargets splits a comma-separated list of IP addresses and
// hostnames and returns the IP addresses to look up, resolving each hostname.
// A hostname that does not resolve is reported in the returned errors instead,
// with the resolver's own error only in the log, since it names the DNS server.
// It stops once ctx's deadline has passed; the caller then answers 504.
func (s *Server) resolveLookupTargets(
ctx context.Context, target string,
) ([]string, []string) {
var ipsToLookup, resolveErrors []string
for t := range strings.SplitSeq(target, ",") {
t = strings.TrimSpace(t)
@@ -839,36 +808,6 @@ func splitLookupTargets(target string) ([]string, error) {
continue
}
if len(targets) == maxLookupTargets {
return nil, errTooManyTargets
}
targets = append(targets, t)
}
return targets, nil
}
// resolveLookupTargets splits a comma-separated list of IP addresses and
// hostnames and returns the IP addresses to look up, resolving each hostname.
// A hostname that does not resolve is reported in the returned errors instead,
// with the resolver's own error only in the log, since it names the DNS server.
// Only the public addresses a hostname resolves to are kept, so the answer
// does not show the deployment's private addresses; a hostname with none is
// reported in the errors. More than maxLookupTargets entries, or addresses once
// resolved, return errTooManyTargets. It stops once ctx's deadline has passed;
// the caller then answers 504.
func (s *Server) resolveLookupTargets(
ctx context.Context, target string,
) ([]string, []string, error) {
targets, err := splitLookupTargets(target)
if err != nil {
return nil, nil, err
}
var ipsToLookup, resolveErrors []string
for _, t := range targets {
// Check if this target is an IP address
if net.ParseIP(t) != nil {
ipsToLookup = append(ipsToLookup, t)
@@ -877,9 +816,9 @@ func (s *Server) resolveLookupTargets(
}
// It's a hostname - resolve it
resolved, err := s.resolver.LookupHost(ctx, t)
resolved, err := net.DefaultResolver.LookupHost(ctx, t)
if deadlinePassed(ctx) {
return nil, nil, nil
return nil, nil
}
if err != nil {
@@ -889,45 +828,16 @@ func (s *Server) resolveLookupTargets(
continue
}
public := publicAddresses(resolved)
if len(public) == 0 {
resolveErrors = append(resolveErrors, t+": "+hostNotPublicMessage)
continue
}
ipsToLookup = append(ipsToLookup, public...)
ipsToLookup = append(ipsToLookup, resolved...)
}
if len(ipsToLookup) > maxLookupTargets {
return nil, nil, errTooManyTargets
}
return ipsToLookup, resolveErrors, nil
}
// publicAddresses returns the addresses in addrs that are global unicast and
// not private; Go counts the private ranges as global unicast.
func publicAddresses(addrs []string) []string {
var public []string
for _, a := range addrs {
ip, err := netip.ParseAddr(a)
if err == nil && ip.IsGlobalUnicast() && !ip.IsPrivate() {
public = append(public, a)
}
}
return public
return ipsToLookup, resolveErrors
}
// lookupPTRNames returns the PTR records of ip without their trailing dots, or
// nil when the lookup fails or runs past ptrLookupTimeout.
func (s *Server) lookupPTRNames(ctx context.Context, ip string) []string {
ctx, cancel := context.WithTimeout(ctx, ptrLookupTimeout)
defer cancel()
ptrs, err := s.resolver.LookupAddr(ctx, ip)
// nil when the lookup fails.
func lookupPTRNames(ctx context.Context, ip string) []string {
ptrs, err := net.DefaultResolver.LookupAddr(ctx, ip)
if err != nil {
return nil
}
+3 -323
View File
@@ -9,9 +9,7 @@ import (
"net/http/httptest"
"runtime"
"slices"
"strconv"
"strings"
"sync/atomic"
"testing"
"time"
@@ -808,7 +806,8 @@ func (failingDB) GetPrefixDetailsContext(
// the health check with a store whose reads fail with the driver's error text,
// and a prefix that does not parse, and checks that no answer carries the
// driver's or the prefix parser's text. Before the fix the answers sent it to
// the client (https://git.eeqj.de/sneak/routewatch/issues/108).
// the client (https://git.eeqj.de/sneak/routewatch/issues/108). The IP lookup
// answers 404, as it does whenever no address could be looked up.
func TestJSONRoutesDoNotSendErrorDetails(t *testing.T) {
t.Parallel()
@@ -824,7 +823,7 @@ func TestJSONRoutesDoNotSendErrorDetails(t *testing.T) {
{statsPath, http.StatusInternalServerError},
{"/api/v1/as/64500", http.StatusInternalServerError},
{"/api/v1/prefix/192.0.2.0/24", http.StatusInternalServerError},
{"/api/v1/ip/192.0.2.1", http.StatusInternalServerError},
{"/api/v1/ip/192.0.2.1", http.StatusNotFound},
{"/api/v1/prefix/foo/24", http.StatusBadRequest},
}
@@ -847,325 +846,6 @@ func TestJSONRoutesDoNotSendErrorDetails(t *testing.T) {
}
}
// The addresses failingIPLookupDB fails the read of and finds a route for.
const (
failingLookupIP = "192.0.2.1"
routedLookupIP = "192.0.2.2"
)
// failingIPLookupDB embeds database.Store (left nil) and answers the IP lookup
// with errDiskIO for failingLookupIP, a route for routedLookupIP, and no route
// for any other address. Every other Store method is unused and would panic if
// called.
type failingIPLookupDB struct {
database.Store
}
func (failingIPLookupDB) GetIPInfoContext(
_ context.Context, ip string,
) (*database.IPInfo, error) {
switch ip {
case failingLookupIP:
return nil, errDiskIO
case routedLookupIP:
return &database.IPInfo{IP: ip}, nil
default:
return nil, database.ErrNoRoute
}
}
// TestIPLookupAnswers500WhenDatabaseReadFails sends IP lookups through the
// server's router with a store whose read of one address fails with the
// driver's error. A lookup that asks for that address answers 500 with the
// JSON error body, as the AS and prefix JSON routes do; an address with no
// route still answers 404. Before the fix the failed read was reported as if
// the address had no route: 404 for that address alone, and 200 beside a
// routed address (https://git.eeqj.de/sneak/routewatch/issues/109).
func TestIPLookupAnswers500WhenDatabaseReadFails(t *testing.T) {
t.Parallel()
s := New(failingIPLookupDB{}, streamer.New(logger.New(), metrics.New()),
logger.New(), &config.Config{})
tests := []struct {
path string
wantCode int
}{
{
path: "/api/v1/ip/" + failingLookupIP,
wantCode: http.StatusInternalServerError,
},
{
path: "/api/v1/ip/" + failingLookupIP + "," + routedLookupIP,
wantCode: http.StatusInternalServerError,
},
{
path: "/api/v1/ip/203.0.113.1",
wantCode: http.StatusNotFound,
},
}
for _, tt := range tests {
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, tt.path, nil))
if rec.Code != tt.wantCode {
t.Errorf("%s: status %d, want %d; body %s",
tt.path, rec.Code, tt.wantCode, rec.Body)
}
checkJSONErrorCode(t, tt.path, rec, tt.wantCode)
}
}
// errNoSuchHost is what fakeResolver answers a lookup it has no answer for
// with.
var errNoSuchHost = errors.New("no such host")
// fakeResolver answers a host name lookup with the addresses hosts lists for
// the name, and a PTR lookup with no names or, when ptrNeverAnswers is set,
// not at all: it waits until the lookup's context ends. hostLookups counts the
// host name lookups.
type fakeResolver struct {
hosts map[string][]string
ptrNeverAnswers bool
hostLookups atomic.Int64
}
func (r *fakeResolver) LookupHost(_ context.Context, host string) ([]string, error) {
r.hostLookups.Add(1)
addrs, ok := r.hosts[host]
if !ok {
return nil, errNoSuchHost
}
return addrs, nil
}
func (r *fakeResolver) LookupAddr(ctx context.Context, _ string) ([]string, error) {
if r.ptrNeverAnswers {
<-ctx.Done()
return nil, ctx.Err()
}
return nil, errNoSuchHost
}
// countingIPLookupDB embeds database.Store (left nil) and counts its IP
// lookups, answering each that the address has no route. Every other Store
// method is unused and would panic if called.
type countingIPLookupDB struct {
database.Store
lookups atomic.Int64
}
func (d *countingIPLookupDB) GetIPInfoContext(
_ context.Context, _ string,
) (*database.IPInfo, error) {
d.lookups.Add(1)
return nil, database.ErrNoRoute
}
// TestIPLookupRefusesTooManyAddresses sends IP lookups of maxLookupTargets
// addresses and of one more, listed or as a host name that resolves to them,
// and a lookup listing maxLookupTargets+1 host names. The longer lookups answer
// 400 without reading the database, and the list of host names without
// resolving any of them; the others read the database once per address. Before
// the fix every address was read and every name resolved, however many there
// were (https://git.eeqj.de/sneak/routewatch/issues/110).
func TestIPLookupRefusesTooManyAddresses(t *testing.T) {
t.Parallel()
addrs := make([]string, maxLookupTargets+1)
names := make([]string, maxLookupTargets+1)
for i := range addrs {
addrs[i] = "192.0.2." + strconv.Itoa(i+1)
names[i] = "host" + strconv.Itoa(i+1) + ".test"
}
hosts := map[string][]string{
"most.test": addrs[:maxLookupTargets],
"too-many.test": addrs,
}
tests := []struct {
target string
wantCode int
wantLookups int64
wantHostLookups int64
}{
{
strings.Join(addrs[:maxLookupTargets], ","),
http.StatusNotFound, maxLookupTargets, 0,
},
{strings.Join(addrs, ","), http.StatusBadRequest, 0, 0},
{"most.test", http.StatusNotFound, maxLookupTargets, 1},
{"too-many.test", http.StatusBadRequest, 0, 1},
{strings.Join(names, ","), http.StatusBadRequest, 0, 0},
}
for _, tt := range tests {
db := &countingIPLookupDB{}
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(),
&config.Config{})
resolver := &fakeResolver{hosts: hosts}
s.resolver = resolver
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, "/api/v1/ip/"+tt.target, nil))
if rec.Code != tt.wantCode {
t.Errorf("%s: status %d, want %d; body %s",
tt.target, rec.Code, tt.wantCode, rec.Body)
}
checkJSONErrorCode(t, tt.target, rec, tt.wantCode)
got := db.lookups.Load()
if got != tt.wantLookups {
t.Errorf("%s: %d database reads, want %d",
tt.target, got, tt.wantLookups)
}
gotHostLookups := resolver.hostLookups.Load()
if gotHostLookups != tt.wantHostLookups {
t.Errorf("%s: %d host name lookups, want %d",
tt.target, gotHostLookups, tt.wantHostLookups)
}
}
}
// addrInTestPrefixV4 is an address in testPrefixV4, which
// newServerWithTestRoutes stores a route for.
const addrInTestPrefixV4 = "192.0.2.1"
// TestIPLookupKeepsRouteWhenPTRLookupDoesNotAnswer looks up a routed address
// with a resolver that never answers its PTR lookup. Once the PTR lookup's own
// limit passes, the lookup answers 200 with the route. Before the fix the PTR
// lookup ran until the request's time limit, and the lookup answered 504
// without the route (https://git.eeqj.de/sneak/routewatch/issues/110).
func TestIPLookupKeepsRouteWhenPTRLookupDoesNotAnswer(t *testing.T) {
t.Parallel()
s := newServerWithTestRoutes(t)
s.resolver = &fakeResolver{ptrNeverAnswers: true}
// The request's time limit passes after the PTR lookup's.
ctx, cancel := context.WithTimeout(t.Context(), 2*ptrLookupTimeout)
defer cancel()
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
ctx, http.MethodGet, "/api/v1/ip/"+addrInTestPrefixV4, nil))
if rec.Code != http.StatusOK {
t.Fatalf("status %d, want %d; body %s", rec.Code, http.StatusOK, rec.Body)
}
var body struct {
Data IPLookupResponse `json:"data"`
}
err := json.Unmarshal(rec.Body.Bytes(), &body)
if err != nil {
t.Fatalf("decoding the answer %q: %v", rec.Body, err)
}
if len(body.Data.Results) != 1 || body.Data.Results[0].Netblock != testPrefixV4 {
t.Errorf("answer %s, want one result in %s", rec.Body, testPrefixV4)
}
}
// The host name TestIPLookupHidesPrivateAddresses resolves to a private
// address, and that address.
const (
internalHostName = "internal.test"
internalHostAddr = "10.0.0.5"
)
// TestIPLookupHidesPrivateAddresses looks up host names that resolve to a
// private address, alone or beside a routed address. No answer carries the
// private address, and a name with no other address is reported as having no
// public address, in the 400 when nothing else is left to look up. Before the
// fix the private address was looked up and named in the answer, "No routes
// found: 10.0.0.5: no route found for IP 10.0.0.5", so the API showed what the
// deployment's own DNS knows (https://git.eeqj.de/sneak/routewatch/issues/110).
func TestIPLookupHidesPrivateAddresses(t *testing.T) {
t.Parallel()
s := newServerWithTestRoutes(t)
s.resolver = &fakeResolver{hosts: map[string][]string{
internalHostName: {internalHostAddr},
"mixed.test": {internalHostAddr, addrInTestPrefixV4},
}}
notPublic := internalHostName + ": " + hostNotPublicMessage
tests := []struct {
target string
wantCode int
wantMessage string
wantErrors []string
}{
{
internalHostName, http.StatusBadRequest,
"No valid IPs or hostnames provided: " + notPublic, nil,
},
{
internalHostName + "," + addrInTestPrefixV4, http.StatusOK, "",
[]string{notPublic},
},
{"mixed.test", http.StatusOK, "", nil},
}
for _, tt := range tests {
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, "/api/v1/ip/"+tt.target, nil))
if rec.Code != tt.wantCode {
t.Errorf("%s: status %d, want %d; body %s",
tt.target, rec.Code, tt.wantCode, rec.Body)
}
if strings.Contains(rec.Body.String(), internalHostAddr) {
t.Errorf("%s: body %s carries %s", tt.target, rec.Body, internalHostAddr)
}
var body struct {
Data IPLookupResponse `json:"data"`
Error struct {
Msg string `json:"msg"`
} `json:"error"`
}
err := json.Unmarshal(rec.Body.Bytes(), &body)
if err != nil {
t.Errorf("%s: decoding the answer %q: %v", tt.target, rec.Body, err)
continue
}
if body.Error.Msg != tt.wantMessage {
t.Errorf("%s: error message %q, want %q",
tt.target, body.Error.Msg, tt.wantMessage)
}
if !slices.Equal(body.Data.Errors, tt.wantErrors) {
t.Errorf("%s: errors %q, want %q",
tt.target, body.Data.Errors, tt.wantErrors)
}
}
}
// settledGoroutineCount lets transient goroutines finish, then reports the
// current count.
func settledGoroutineCount() int {
+3 -9
View File
@@ -142,9 +142,8 @@ func deadlinePassed(ctx context.Context) bool {
//
// If a response is not valid JSON or is empty when JSON is expected, the middleware
// returns a properly formatted JSON error response. If the request's time limit
// has passed, it answers 504 with the error message "Request timeout". A 4xx
// keeps its status, with the status's standard text as the message. For other
// errors, it returns "Internal server error", with status 500 if the original
// has passed, it answers 504 with the error message "Request timeout". For other
// errors, it returns "Internal server error" with status 500 if the original
// status was 200.
//
// Non-JSON responses are passed through unchanged.
@@ -204,13 +203,8 @@ func JSONValidationMiddleware(next http.Handler) http.Handler {
w.WriteHeader(statusCode)
errorMsg := internalErrorMessage
switch {
case statusCode == http.StatusGatewayTimeout:
if statusCode == http.StatusGatewayTimeout {
errorMsg = requestTimeoutMessage
case statusCode >= http.StatusBadRequest &&
statusCode < http.StatusInternalServerError:
errorMsg = http.StatusText(statusCode)
}
response := map[string]any{
@@ -58,34 +58,6 @@ func TestResponsesCarrySecurityHeaders(t *testing.T) {
}
}
// TestJSONValidationKeeps4xxWithoutBody wraps a handler that answers 400
// without a body in JSONValidationMiddleware and checks that the JSON error
// body keeps the 400 and gives the status's own text. Before the fix it said
// "Internal server error"
// (https://git.eeqj.de/sneak/routewatch/issues/111).
func TestJSONValidationKeeps4xxWithoutBody(t *testing.T) {
t.Parallel()
handler := JSONValidationMiddleware(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusBadRequest)
}))
rec := httptest.NewRecorder()
handler.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, "/", nil))
const name = "400 without a body"
if rec.Code != http.StatusBadRequest {
t.Errorf("%s: status %d, want %d; body %s",
name, rec.Code, http.StatusBadRequest, rec.Body)
}
checkJSONErrorCode(t, name, rec, http.StatusBadRequest)
checkJSONErrorMessage(t, name, rec, http.StatusText(http.StatusBadRequest))
}
// TestPagesCarryTheCSPNonce checks that every script and style block of each
// page carries the nonce its Content-Security-Policy header names, so that the
// browser runs and applies them, that each answer has a nonce of its own, and
+1 -34
View File
@@ -1,33 +1,18 @@
package server
import (
"net/http"
"time"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
)
// The error messages the JSON routes under /api/v1 and /ip answer with when no
// route matches the path, and when none takes the method.
const (
notFoundMessage = "Not found"
methodNotAllowedMessage = "Method not allowed"
// jsonRoutesAllow is the Allow header of their 405 answer: every one of
// them takes GET and HEAD, and no other method.
jsonRoutesAllow = "GET, HEAD"
)
// setupRoutes configures the HTTP routes
func (s *Server) setupRoutes() {
r := chi.NewRouter()
// Middleware
r.Use(DecodedPathMiddleware)
// GetHead serves HEAD with the GET handler. It looks the route up by the
// path DecodedPathMiddleware sets, so it comes after it.
r.Use(middleware.GetHead)
r.Use(SecurityHeadersMiddleware)
r.Use(middleware.RequestID)
r.Use(RealIPMiddleware(s.trustedProxies))
@@ -55,34 +40,16 @@ func (s *Server) setupRoutes() {
r.Get("/prefixlength/{length}", s.handlePrefixLength())
r.Get("/prefixlength6/{length}", s.handlePrefixLength6())
// The JSON routes answer a path or method they do not take with the JSON
// error body, as they answer everything else.
jsonNotFound := func(w http.ResponseWriter, _ *http.Request) {
s.writeJSONError(w, http.StatusNotFound, notFoundMessage)
}
jsonMethodNotAllowed := func(w http.ResponseWriter, _ *http.Request) {
// chi's own 405 answer sets Allow; one set in its place must too.
w.Header().Set("Allow", jsonRoutesAllow)
s.writeJSONError(w, http.StatusMethodNotAllowed, methodNotAllowedMessage)
}
// IP info JSON endpoints (replaces old /ip redirect)
r.Route("/ip", func(r chi.Router) {
r.Use(JSONValidationMiddleware)
r.NotFound(jsonNotFound)
r.MethodNotAllowed(jsonMethodNotAllowed)
r.Get("/", s.handleIPInfo()) // Client IP
// GetHead does not reach this route: mounting the group registers
// /ip and /ip/ for every method, so HEAD arrives here as HEAD.
r.Head("/", s.handleIPInfo())
r.Get("/", s.handleIPInfo()) // Client IP
r.Get("/{addr}", s.handleIPInfo()) // Specified IP
})
// API routes
r.Route("/api/v1", func(r chi.Router) {
r.Use(JSONValidationMiddleware)
r.NotFound(jsonNotFound)
r.MethodNotAllowed(jsonMethodNotAllowed)
r.Get("/stats", s.handleStats())
r.Get("/ip/{ip}", s.handleIPLookup())
r.Get("/as/{asn}", s.handleASDetailJSON())
-100
View File
@@ -1,100 +0,0 @@
package server
import (
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
)
// TestRoutesAnswerHEAD sends HEAD for a page and JSON routes through the
// server's router and checks that each answers as GET does. /ip and /ip/, the
// lookup of the caller's own address, are in the list because chi's GetHead
// does not reach them. Before the fix every route answered HEAD with 405
// (https://git.eeqj.de/sneak/routewatch/issues/111).
func TestRoutesAnswerHEAD(t *testing.T) {
t.Parallel()
s := newServerWithTestRoutes(t)
for _, path := range []string{"/", statsPath, "/ip", "/ip/"} {
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodHead, path, nil))
if rec.Code != http.StatusOK {
t.Errorf("HEAD %s: status %d, want %d; body %s",
path, rec.Code, http.StatusOK, rec.Body)
}
}
}
// TestJSONRoutesAnswerUnmatchedRequestsWithJSONErrors sends a method the JSON
// routes do not take, and a path none of them matches, to each group of JSON
// routes through the server's router. Each must answer with the JSON error
// body, and a 405 must name the methods the routes take in its Allow header.
// Before the fix a 405 said "Internal server error" and a 404 was chi's plain
// text answer (https://git.eeqj.de/sneak/routewatch/issues/111).
func TestJSONRoutesAnswerUnmatchedRequestsWithJSONErrors(t *testing.T) {
t.Parallel()
s := newServerWithTestRoutes(t)
tests := []struct {
method string
path string
wantCode int
wantMsg string
}{
{http.MethodPost, statsPath, http.StatusMethodNotAllowed, methodNotAllowedMessage},
{http.MethodPost, "/ip/", http.StatusMethodNotAllowed, methodNotAllowedMessage},
{http.MethodGet, "/api/v1/nothing", http.StatusNotFound, notFoundMessage},
{http.MethodGet, "/ip/192.0.2.1/nothing", http.StatusNotFound, notFoundMessage},
}
for _, tt := range tests {
name := tt.method + " " + tt.path
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), tt.method, tt.path, nil))
if rec.Code != tt.wantCode {
t.Errorf("%s: status %d, want %d; body %s",
name, rec.Code, tt.wantCode, rec.Body)
}
checkJSONErrorCode(t, name, rec, tt.wantCode)
checkJSONErrorMessage(t, name, rec, tt.wantMsg)
allow := rec.Header().Get("Allow")
if tt.wantCode == http.StatusMethodNotAllowed && allow != jsonRoutesAllow {
t.Errorf("%s: Allow %q, want %q", name, allow, jsonRoutesAllow)
}
}
}
// checkJSONErrorMessage checks that rec holds the JSON error body with the
// given message.
func checkJSONErrorMessage(
t *testing.T, name string, rec *httptest.ResponseRecorder, want string,
) {
t.Helper()
var body struct {
Error struct {
Msg string `json:"msg"`
} `json:"error"`
}
err := json.Unmarshal(rec.Body.Bytes(), &body)
if err != nil {
t.Errorf("%s: decoding the answer %q: %v", name, rec.Body, err)
return
}
if body.Error.Msg != want {
t.Errorf("%s: error message %q, want %q", name, body.Error.Msg, want)
}
}
+1 -14
View File
@@ -32,13 +32,6 @@ type ASNFetcher interface {
GetStats() ASNFetcherStats
}
// dnsResolver looks up the addresses of host names and the names of addresses
// in DNS, as *net.Resolver does.
type dnsResolver interface {
LookupHost(ctx context.Context, host string) ([]string, error)
LookupAddr(ctx context.Context, addr string) ([]string, error)
}
// Server provides HTTP endpoints for status monitoring
type Server struct {
router *chi.Mux
@@ -52,10 +45,6 @@ type Server struct {
// trustedProxies are the reverse proxies whose X-Forwarded-For header
// RealIPMiddleware believes
trustedProxies []netip.Prefix
// resolver makes the IP lookup's DNS lookups: net.DefaultResolver, or a
// fake in tests
resolver dnsResolver
}
// New creates a new HTTP server
@@ -71,7 +60,6 @@ func New(
logger: logger,
port: cfg.Port,
trustedProxies: cfg.TrustedProxies,
resolver: net.DefaultResolver,
}
s.setupRoutes()
@@ -139,8 +127,7 @@ func (s *Server) Stop(ctx context.Context) error {
return nil
}
// SetASNFetcher sets the ASN WHOIS fetcher for on-demand lookups. Call it
// before Start: the handlers read the fetcher without a lock.
// SetASNFetcher sets the ASN WHOIS fetcher for on-demand lookups.
func (s *Server) SetASNFetcher(fetcher ASNFetcher) {
s.asnFetcher = fetcher
}
+5 -5
View File
@@ -106,12 +106,12 @@ type RawMessageHandler func(line string)
// handlerMetrics tracks performance metrics for a handler
type handlerMetrics struct {
processedCount uint64 // Total messages processed
droppedCount atomic.Uint64 // Total messages dropped; not under mu
droppedCount uint64 // Total messages dropped
totalTime time.Duration // Total processing time (for average calculation)
minTime time.Duration // Minimum processing time
maxTime time.Duration // Maximum processing time
queueHighWaterMark int // Maximum queue length seen
mu sync.Mutex // Protects the other metrics
mu sync.Mutex // Protects the metrics
}
// handlerInfo wraps a handler with its queue and metrics
@@ -371,7 +371,7 @@ func (s *Streamer) GetHandlerStats() []HandlerStats {
QueueCapacity: cap(info.queue),
QueueHighWaterMark: info.metrics.queueHighWaterMark,
ProcessedCount: info.metrics.processedCount,
DroppedCount: info.metrics.droppedCount.Load(),
DroppedCount: info.metrics.droppedCount,
MinProcessTime: info.metrics.minTime,
MaxProcessTime: info.metrics.maxTime,
}
@@ -488,7 +488,7 @@ func (s *Streamer) logMetrics() {
"queue_len", len(info.queue),
"queue_cap", cap(info.queue),
"processed", info.metrics.processedCount,
"dropped", info.metrics.droppedCount.Load(),
"dropped", info.metrics.droppedCount,
"avg_time", avgTime,
"min_time", info.metrics.minTime,
"max_time", info.metrics.maxTime,
@@ -858,7 +858,7 @@ func (s *Streamer) dispatchMessage(
}
if !s.queueMessage(info, msg) {
info.metrics.droppedCount.Add(1)
atomic.AddUint64(&info.metrics.droppedCount, 1)
s.totalDropped.Add(1)
continue
+1 -86
View File
@@ -1,12 +1,9 @@
package streamer
import (
"bytes"
"context"
"encoding/json"
"errors"
"io"
"log/slog"
"net/http"
"net/http/httptest"
"runtime"
@@ -419,8 +416,7 @@ func TestStopQueuesTheHeldStateMessages(t *testing.T) {
// and a STATE message held for it, and a worker that stays busy until the test
// ends, as one does whose handler waits for a database write. Stop must return
// within its deadline plus a small margin instead of waiting for room on the
// queue and for the worker, and log as not handed over the held message and
// every message on the queue.
// queue and for the worker.
func TestStopReturnsByItsDeadline(t *testing.T) {
t.Parallel()
@@ -429,15 +425,10 @@ func TestStopReturnsByItsDeadline(t *testing.T) {
margin = 2 * time.Second
)
var logs bytes.Buffer
s := newStreamerWithFullStateQueue("")
s.logger = &logger.Logger{Logger: slog.New(slog.NewJSONHandler(&logs,
&slog.HandlerOptions{Level: slog.LevelError}))}
s.handlers[0].heldState = []*ristypes.RISMessage{
{Type: ristypes.MessageTypeState},
}
wantNotHandedOver := 1 + len(s.handlers[0].queue)
// Stop cancels the stream Start would have started; there is none here.
s.cancel = func() {}
@@ -461,20 +452,6 @@ func TestStopReturnsByItsDeadline(t *testing.T) {
case <-time.After(deadline + margin):
t.Fatalf("Stop did not return within %s of its deadline", margin)
}
var logged struct {
Messages int `json:"messages"`
}
err := json.Unmarshal(logs.Bytes(), &logged)
if err != nil {
t.Fatalf("want the error Stop logs, got %q: %v", logs.String(), err)
}
if logged.Messages != wantNotHandedOver {
t.Errorf("Stop logged %d messages not handed over, want %d",
logged.Messages, wantNotHandedOver)
}
}
// TestStopBeforeMessageReachesQueues stops the streamer after the read loop
@@ -621,68 +598,6 @@ func TestStreamReadsLongLine(t *testing.T) {
}
}
// blockedHandler wants UPDATE messages, has room on its queue for one, and
// does not return from HandleMessage until release is closed.
type blockedHandler struct {
release chan struct{}
}
func (blockedHandler) WantsMessage(messageType string) bool {
return messageType == ristypes.MessageTypeUpdate
}
func (blockedHandler) QueueCapacity() int { return 1 }
func (h blockedHandler) HandleMessage(*ristypes.RISMessage) { <-h.release }
// TestHandlerStatsReadWhileMessagesAreDropped has the server send a few UPDATE
// lines to a blockedHandler, so the read loop drops all but the one the
// handler holds and the one on its queue, while the test reads the handler's
// statistics. The drop count must reach the number dropped, and the race
// detector must find the count read and written safely.
func TestHandlerStatsReadWhileMessagesAreDropped(t *testing.T) {
t.Parallel()
const (
lines = 5
wantDropped = lines - 2
maxWait = 5 * time.Second
)
srv := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
_, _ = io.WriteString(w, strings.Repeat(updateLine, lines))
flush(w)
<-r.Context().Done()
}))
defer srv.Close()
s := New(logger.New(), metrics.New())
s.url = srv.URL
handler := blockedHandler{release: make(chan struct{})}
s.RegisterHandler(handler)
err := s.Start(t.Context())
if err != nil {
t.Fatalf("Start returned error: %v", err)
}
defer s.Stop(t.Context())
defer close(handler.release)
// The test polls instead of waiting for a signal from the read loop: a
// signal would order the drops before the reads here, and the race
// detector would have nothing to report.
deadline := time.Now().Add(maxWait)
for s.GetHandlerStats()[0].DroppedCount < wantDropped {
if time.Now().After(deadline) {
t.Fatalf("fewer than %d messages counted as dropped within %s",
wantDropped, maxWait)
}
}
}
// TestPeerCountCountsEveryPeerHeardFrom has the server send an UPDATE from
// stateDownPeer and a KEEPALIVE from another peer, with no OPEN from either,
// and then a message that ends stateDownPeer's session: a NOTIFICATION, or a
-54
View File
@@ -1,54 +0,0 @@
package templates_test
import (
"bytes"
"html/template"
"strings"
"testing"
"git.eeqj.de/sneak/routewatch/internal/templates"
)
// TestFooterLicense renders the home and status pages and expects each footer
// to name the licence in LICENSE, MIT, and not WTFPL.
func TestFooterLicense(t *testing.T) {
t.Parallel()
const (
license = "MIT"
wrongLicense = "WTFPL"
)
pages := map[string]*template.Template{
"index": templates.IndexTemplate(),
"status": templates.StatusTemplate(),
}
for name, tmpl := range pages {
t.Run(name, func(t *testing.T) {
t.Parallel()
var page bytes.Buffer
err := tmpl.Execute(&page, struct{ CSPNonce string }{})
if err != nil {
t.Fatalf("rendering the page: %v", err)
}
_, footer, found := strings.Cut(page.String(), "<footer")
if !found {
t.Fatal("the page has no footer")
}
footer, _, _ = strings.Cut(footer, "</footer>")
if !strings.Contains(footer, license) {
t.Errorf("footer does not name %s: %q", license, footer)
}
if strings.Contains(footer, wrongLicense) {
t.Errorf("footer names %s: %q", wrongLicense, footer)
}
})
}
}
+2 -2
View File
@@ -19,8 +19,8 @@ const (
Author = "@sneak"
// AuthorURL is the author's website
AuthorURL = "https://sneak.berlin"
// License is the program license, as LICENSE grants it
License = "MIT"
// License is the program license
License = "WTFPL"
// RepoURL is the git repository URL
RepoURL = "https://git.eeqj.de/sneak/routewatch"
)