sneak/routewatch
1
0
Fork 0
Code Issues 12 Pull Requests 5 Actions 50 Packages Projects Releases Wiki Activity

Compare commits

base: sneak/routewatch:d15a5e91b949bd447afa066f035d62a2efecb7db
Branches Tags
sneak/routewatch:main sneak/routewatch:issue-52-revendor-prompts sneak/routewatch:issue-66-prefix-length-sample sneak/routewatch:issue-55-route-per-prefix-and-peer sneak/routewatch:issue-65-batch-write-off-mutex sneak/routewatch:next sneak/routewatch:issue-70-timeout-504 sneak/routewatch:issue-27-cached-stats sneak/routewatch:prod
...
compare: sneak/routewatch:issue-52-revendor-prompts
Branches Tags
sneak/routewatch:issue-52-revendor-prompts sneak/routewatch:issue-66-prefix-length-sample sneak/routewatch:issue-55-route-per-prefix-and-peer sneak/routewatch:issue-65-batch-write-off-mutex sneak/routewatch:next sneak/routewatch:issue-70-timeout-504 sneak/routewatch:main sneak/routewatch:issue-27-cached-stats sneak/routewatch:prod
136 Commits
d15a5e91b9 ... issue-52-revendor-prompts
Author SHA1 Message Date
sneak 793c75751d Re-vendor the shared files from sneak/prompts at dd4027b (closes #52)
check / check (push) Waiting to run
Details 
Takes .golangci.yml, .dockerignore, .gitignore, .editorconfig, the
workflow and REPO_POLICIES.md from sneak/prompts dd4027b, keeping this
repository's own entries after the shared content. Lint (golangci-lint
v2.14.0) and test become Dockerfile phases that script/lint and
script/test build alone and uncached; the image build depends on both,
and script/cibuild bootstraps, runs script/check, then builds the image.
The agent guidance file moves to AGENTS.md. The Go code is fixed for
what the new configuration finds; reads that iterate rows now also
report an iteration error.

Model: opus-5-5
 2026-10-14 20:03:00 +00:00
clawbot e0b9c2685f Reconnect when RIS Live ends or stalls the stream (closes #53)
check / check (push) Waiting to run
Details 
When RIS Live ended the response normally, the reconnect loop stopped for good and the health check kept reporting the feed connected; a connection that stayed open but sent nothing blocked forever. Now only `Stop` ends the loop: any other end clears the connected flag and reconnects with the existing backoff. If no line arrives for 60 seconds, including while waiting for RIS Live to answer the request, the streamer drops the connection and reconnects. Tests use a local fake server. README "Reconnection" describes both.

Model: opus-5-5
 2026-10-14 19:42:57 +02:00
clawbot ad4c3d91f1 Show the prefix page's origin names from the prefix query (closes #67)
check / check (push) Waiting to run
Details 
The prefix page looked up each origin AS's details only to show its handle and description, and that lookup reads every live route the AS originates. `GetPrefixDetailsContext` already selected the handle and description and dropped them; it now returns them keyed by AS number, and the page uses them. The origin list now also shows the description after the handle, which it never rendered. The JSON prefix answer is unchanged. The test serves the page from a store whose AS lookup always fails.

Model: opus-5-5
 2026-10-14 19:08:58 +02:00
clawbot b4dee48401 Exit with status 1 when the web server cannot listen (closes #73)
check / check (push) Waiting to run
Details 
`Server.Start` opened its port inside a goroutine and returned nil at once, so a port already in use was only logged and the daemon kept running with no web server. It now opens the listener before returning and returns that error; serving runs in the background on the open listener. `CLIEntry` turns an error from `Run` into an fx shutdown with exit status 1, after the normal stop sequence, so a container restart policy can act. A serving error after a successful listen is still only logged.

Model: opus-5-5
 2026-10-14 18:26:04 +02:00
clawbot 616f13071f Parse the prefix on the prefix pages before looking it up (closes #71)
check / check (push) Waiting to run
Details 
`/prefix/{prefix}/{len}` and `/api/v1/prefix/{prefix}/{len}` passed the URL text to the lookup as typed: a prefix that does not parse answered 500, and a valid prefix written in another form (`2001:DB8:0:0::/32`, `192.0.2.1/24`) answered 404. Both now parse the prefix with `net/netip`, mask it and look up the form the tables store; one that does not parse answers 400. The answer shows the stored form of the prefix. The test drives both pages through the router against a real database.

Model: opus-5-5
 2026-10-14 17:24:02 +02:00
clawbot 521c632087 Ignore routes shorter than /8 or /16 in the IP lookup (closes #57)
check / check (push) Waiting to run
Details 
The IP lookup tried every mask length down to /0, so a default route or a very short route such as `2000::/3` was reported as the owner of any address it covered until a more specific route had been seen. The lookup now stops at /8 for IPv4 and /16 for IPv6 (named constants); an address covered only by shorter routes gets "no route found". The tables still store short routes. The test checks each limit and the length one bit shorter.

Model: opus-5-5
 2026-10-14 17:23:51 +02:00
clawbot 77a708d819 Apply each peer's last update for a prefix in a batch (closes #54)
check / check (push) Waiting to run
Details 
The prefix handler kept one update per prefix in each batch across all peers, chosen by a timestamp cut to whole seconds, so most peers' announcements and withdrawals were lost and the first update in a second won. A batch now keeps the last update per prefix and peer in feed order. A message's withdrawals are queued before its announcements, so a prefix both withdrawn and announced in one message keeps its route (RFC 4271 section 4.3). Tests run each case through the handler into a real database and check the in-memory counts.

Model: opus-5-5
 2026-10-14 17:09:13 +02:00
clawbot a43935308e Count every reconnect in reconnect_count (closes #68)
check / check (push) Waiting to run
Details 
`reconnect_count` in `/api/v1/stats` stayed 0 however often the stream reconnected: `SetConnected` counted a reconnect only while the stream was still marked connected, and the streamer always marks it disconnected first. It now counts one reconnect for every connection after the first, recognised by the zero `connectedSince` before the first connection. A test connects and disconnects twice and expects 2.

Model: opus-5-5
 2026-10-14 15:59:15 +02:00
sneak 00e51d284f Merge pull request 'Next milestone: production memory under 5 GiB, ready for upaas' (#6) from next into main
check / check (push) Waiting to run
Details 
Reviewed-on: #6
 2026-10-06 00:34:48 +02:00
clawbot a9b68608c6 Serve the /api/v1/stats prefix distribution from memory (closes #30)
check / check (push) Successful in 3m19s
Details 
The prefix distribution was the last query on the stats path. It read one index entry per live route on every request, so on a large database it took the whole 4-second deadline and /api/v1/stats answered 500.

The distribution now lives next to the in-memory counts: seeded once at startup from the same query, then kept exact by every live-route write. A new route whose prefix had no live route adds one at its mask length; a delete that leaves a prefix with no live route takes one away. Each check is one lookup on the prefix index, within the same write. The single-route delete now runs its delete and that lookup in one transaction.

Model: opus-5-5
 2026-10-03 18:43:58 +02:00
clawbot 44a5f4cdca Create new databases with auto_vacuum incremental (closes #43)
check / check (push) Successful in 2m50s
Details 
SQLite only accepts auto_vacuum before the database file is first written. The connection switched to WAL first, which writes the file, so the PRAGMA auto_vacuum in Initialize came too late and was ignored. The setting now goes in the connection string, which the driver applies on open before the journal mode, and the late PRAGMA is removed.

Vacuum now reads every row PRAGMA incremental_vacuum returns: SQLite frees one page per row, and the single step ExecContext took freed only one page per call.

Tests check that every pooled connection sees auto_vacuum incremental on a new database and that one Vacuum call frees every page left by deleting routes.

Model: opus-5-5
 2026-10-03 17:09:29 +02:00
clawbot 32704c601e Look up IP addresses by prefix at each mask length (closes #48)
check / check (push) Successful in 3m13s
Details 
Looking up an address on /ip/ and /api/v1/ip/ read every IPv6 live route and about half of the IPv4 range index, so on a day-sized database it passed the 30-second request timeout. The lookup is now one function for both families: from the longest mask length down, it looks up the address's network prefix on the existing prefix index, and the first live route wins.

The feed sends IPv6 withdrawals uncompressed, so they never matched the stored compressed prefix and never removed a route. Prefixes are now stored in the text form net/netip prints, so IPv6 withdrawals take effect.

ip_start, ip_end and GetASInfoForIP are removed; nothing read them. A database from before this change must be deleted.

Model: opus-5-5
 2026-10-03 16:04:38 +02:00
clawbot 7322f936e4 Stamp the git tag or short commit in a plain docker build (closes #46)
check / check (push) Successful in 3m6s
Details 
A plain `docker build .` stamped `unknown` into the page footer:
.dockerignore left out .git, and the Dockerfile built without the -X
flags. The context now carries .git without .git/config, which can hold
a credential. The build stage takes the VERSION build argument when one
is given, otherwise `git describe --tags --always`, fails the build if
.git is present and no version comes out, and stamps it together with
the full commit the footer links to. `make build` now uses
`git describe` as well, and script/docker is the current shared copy.

Model: opus-5-5
 2026-10-02 08:17:35 +02:00
clawbot 6422d9fa0c Container makes its data directory usable itself (closes #42)
check / check (push) Successful in 8s
Details 
sneak's standing rule: the container makes its data directory usable itself, with no step on the host. entrypoint.sh now creates /var/lib/berlin.sneak.app.routewatch if it is missing and stops the start when any step fails (set -euo pipefail); before, a failed cd went on to change the ownership of whatever directory the script was in, and a failed chown still started the daemon. Taking ownership of the directory and switching to the routewatch user through setpriv are unchanged. The README's upaas volume line now says only which path to mount.

The empty-directory and other-uid cases were run by hand on the built image with upaas-style bind mounts, not added as an automated test.

Model: opus-5-5
 2026-09-29 12:23:42 +02:00
clawbot 057e0bd9a9 Add a .dockerignore (closes #39)
check / check (push) Successful in 6s
Details 
Adds a .dockerignore, which REPO_POLICIES.md lists among the files every repo has. Until now every docker build sent the whole working tree as its build context, and the image's source archive is made from that context. It now keeps out .git, local build and test output, archives, local databases, .env and a local Go workspace. Every tracked file stays in, so the format check, the linter, the tests, the build and the source archive read the same files as before.

Without .git in the context the binary's build info records no git revision; nothing in routewatch reads it.

Model: opus-5-5
 2026-09-29 07:42:20 +02:00
clawbot 87a3bc115e README: license and author up front; bring TODO.md up to date (closes #38)
check / check (push) Successful in 3m14s
Details 
Brings README.md and TODO.md in line with REPO_POLICIES.md and with what is on next. The README's first line now names routewatch, what it is, its MIT licence and its author, @sneak; the License section says MIT and links LICENSE; a new Author section names @sneak.

TODO.md's Status, Next Step and Future Steps now describe next as it is: it waits for sneak to merge the milestone PR, after which the upaas deploy and the run under a real 5 GiB limit are his. Completed Steps gains a line for each issue that landed without one.

Docs only. make fmt formats only Go here, so the markdown was wrapped by hand.

Model: opus-5-5
 2026-09-29 07:06:30 +02:00
clawbot d28a59023a Add MIT LICENSE (closes #1)
check / check (push) Successful in 4m5s
Details 
Adds LICENSE with the MIT licence sneak chose for routewatch, copied byte for byte from sneak/webhooker's LICENSE: copyright 2026, Jeffrey Paul. The rest of the issue, .editorconfig and the Gitea check workflow, was already on next, and the README already points at LICENSE.

Judgement call: our repos write the copyright holder in several different ways and sneak named no form; webhooker's gives his full name and email address, the most explicit of them.

Model: opus-5-5
 2026-09-29 05:05:35 +02:00
clawbot 1ac24669d3 Stop the streamer without sending on closed queues (closes #34)
check / check (push) Successful in 2m38s
Details 
Stopping the daemon while the RIS Live feed was flowing could panic with "send on closed channel" and skip the rest of the shutdown. Stop cancels the stream and closes the handler queues under the streamer's write lock, but the read loop checked for a stop only before parsing each line. It now checks again under the read lock it already takes just before handing a message to the queues, so it never sends to a closed queue. Stop also clears its cancel function and returns early when there is none, so a second call no longer closes the queues again. A test forces both cases.

Behaviour change: Stop before Start now does nothing.

Model: opus-5-5
 2026-09-28 21:42:33 +02:00
clawbot 6187ac8503 Let the daemon receive docker stop's signal itself (closes #33)
check / check (push) Successful in 2m42s
Details 
entrypoint.sh now switches to the routewatch user (UID 1000) with setpriv instead of runuser. setpriv replaces itself with the daemon, so the daemon is the container's main process and receives docker stop's signal itself. runuser stayed in between, passed the signal on and killed the daemon 2 seconds later, so every stop ended with exit 143. The daemon now gets the whole wait the caller allows, up to its own 60-second limit, and a clean stop exits 0. Taking ownership of the state directory and the MALLOC_ARENA_MAX check still run as root first.

Not fixed here: a stop while the feed is flowing can still panic (#34).

Model: opus-5-5
 2026-09-28 21:08:33 +02:00
clawbot f2a9e90625 Refuse invalid settings at start, health check follows PORT (closes #31)
check / check (push) Successful in 3m17s
Details 
A set but invalid PORT (anything but plain digits from 1 to 65535) or a relative XDG_DATA_HOME now stops the start before the database opens; before, a bad PORT left the daemon running without HTTP. entrypoint.sh refuses a MALLOC_ARENA_MAX that is not a positive whole number, since glibc ignores a bad one silently. The HEALTHCHECK probes the port PORT names, 8080 when unset. PORT is now read in internal/config, so server.New takes the config.

The README gives the real Linux state directory, lists XDG_DATA_HOME, and adds "Running under upaas": port, volume, environment, the 5g memory limit and the health check.

Unverified: the 5g memory limit could not be exercised on the build host.

Model: opus-5-5
 2026-09-28 20:08:42 +02:00
clawbot df9e23d503 Serve /api/v1/stats counts from realtime in-memory counters (closes #27)
check / check (push) Successful in 2m36s
Details 
Realtime in-memory counters seeded at startup and adjusted on every insert, update and delete; no periodic recompute. Independent review passed: #29 (comment)

model: claude-opus-4-8 (implementation and review); merged by claude-fable-5
 2026-09-22 09:41:16 +02:00
clawbot 3898daad4e Add _txlock=immediate so batch writes wait instead of dropping (closes #25)
check / check (push) Failing after 0s
Details 
Batch flush paths read before they write, so a deferred transaction starts
as a reader and must upgrade to the write lock on its first INSERT/UPDATE.
When the background maintainer holds the write lock for a WAL checkpoint,
that upgrade fails immediately with "database is locked" and the busy
timeout does not apply, so the batch is dropped. Adding _txlock=immediate
to the DSN makes every transaction take the write lock at BEGIN, so it
waits up to busy_timeout instead of failing.

A regression test drives batch writes against a running checkpoint loop and
fails with "database is locked" without the change.

Model: opus-4-8
 2026-09-21 20:12:33 +02:00
clawbot 08f060045b Cap glibc malloc arenas to keep non-Go RSS bounded (closes #23)
check / check (push) Successful in 3m44s
Details 
After the earlier memory work, SQLite's live heap is bounded but process
RSS still climbed about 70 MiB/min, nearly all anonymous and outside the
Go runtime and outside SQLite's own accounting, without any SQLITE_NOMEM.
The cause is glibc: the SQLite C library allocates and frees millions of
small page-cache chunks from many threads, and glibc keeps each arena's
freed chunks resident. With arenas uncapped it creates up to eight per
core, so on a large host RSS grows with the core count.

Set MALLOC_ARENA_MAX=2 in the image to bound the retained memory. Writes
are already serialized, so the two-arena cap costs no throughput. Update
the README Memory section.

Model: opus-4-8
 2026-09-21 19:29:31 +02:00
clawbot 658aadbb81 Set GOMEMLIMIT in the image and document the memory budget (closes #13)
check / check (push) Failing after 1s
Details 
Add ENV GOMEMLIMIT=1536MiB to the runtime stage so the Go runtime keeps a
1.5 GiB soft heap limit. runuser preserves it the way it already does
XDG_DATA_HOME, so the routewatch process inherits it.

Add a Memory section to the README describing the budget (Go 1.5 GiB soft,
SQLite 640 MiB pool cache and 1.5 GiB hard heap, ~0.2 GiB other), the
required 5 GiB container limit, how to override GOMEMLIMIT, what happens at
each limit, and the DEBUG=routewatch System stats line. Every sentence
matches the behaviour already merged to next.

Model: opus-4-8
 2026-09-21 17:29:29 +02:00
clawbot efaf79c4e3 Shrink the four handler queues from 100,000 to 20,000 (closes #11)
check / check (push) Successful in 2m52s
Details 
Each handler queue held 100,000 message pointers; a message is retained
until the slowest handler drains it, so all four full was a derived worst
case near 800 MiB. Twenty thousand is about four seconds of feed at peak
and caps that at roughly 160 MiB. The streamer already drops rather than
blocks on a full queue, so the smaller bound is safe.

Batch sizes are unchanged. The largest, asnBatchSize, is 30,000 and now
exceeds its queue, but each queued message contributes every ASN in its
path, and every handler also flushes on its own timer regardless of fill,
so batches still flush and no size change is warranted.

Model: opus-4-8
 2026-09-21 16:29:39 +02:00
clawbot 63d62b7bc1 Fix two goroutine leaks: stats handlers on timeout, streamer tickers on reconnect (closes #12)
check / check (push) Failing after 1s
Details 
The stats handlers ran the database query in a goroutine that sent on an
unbuffered channel. When the 4s request timeout won, nothing received and
the goroutine blocked forever; the status page polls every 2s, so once the
query exceeds the timeout every poll leaked one goroutine. Give both
channels capacity 1 so the send always completes.

The streamer started two ticker goroutines per connection that exited only
with the streamer's lifetime context, leaking two on every reconnect. Scope
them to a per-connection context cancelled when the stream call returns.

Tests force the stats timeout repeatedly and drive many reconnects, then
assert the goroutine count settles back to its starting value. The streamer
gains an internal endpoint field so a test can point it at a local server.

Model: opus-4-8
 2026-09-21 16:29:32 +02:00
clawbot cb2374f033 Stop decoding unused Community and Raw fields of RIS messages (closes #9)
check / check (push) Failing after 1s
Details 
Community and Raw are decoded from every live message and read by no
handler; Raw is the hex of the whole BGP message. Each parsed message
sits in up to four handler queues, so retaining them is a large share of
queue memory. Tagging both json:"-" keeps them out of the decoded
message while the fields handlers use still decode. A decode test over
docs/message-examples.json confirms both stay empty and the used fields
(Path, Announcements) still populate.

Model: opus-4-8
 2026-09-21 16:12:41 +02:00
clawbot 6d8ae7f592 Add .editorconfig and Gitea CI workflow (closes #14)
check / check (push) Failing after 1s
Details 
Copy .editorconfig verbatim from sneak/dnswatcher (4-space indents,
LF, trailing-whitespace trim, final newline; tabs for the Makefile).

Add .gitea/workflows/check.yml mirroring dnswatcher: it runs
script/cibuild on push, which builds the Dockerfile that runs
make check, so CI gates every push. The checkout action is pinned by
commit SHA as that repo pins it. No LICENSE is added; that part of
the parent issue awaits an owner decision.

Model: opus-4-8
 2026-09-21 16:01:14 +02:00
clawbot 211fdad9c0 Bound peering AS-path map and swap it instead of copying (closes #10) 
processPeerings now takes the accumulated AS-path map under the lock and
replaces it with a fresh empty one, so each path is processed once and the
map never grows past a single 30-second interval's traffic. HandleMessage
stops adding new paths once maxTrackedPaths (500000) is reached and counts
the drops, which are logged with each run. The 30-minute prune and its
ticker are removed as dead code under the swap, and the map mutex drops
from RWMutex to Mutex since the read path is gone. RecordPeering already
upserts last_seen, so stored peerings are unchanged.

Model: opus-4-8
 2026-09-21 16:01:08 +02:00
clawbot 594e7a504b Bound SQLite memory across the whole connection pool (closes #8) 
The 3 GiB page cache and temp_store=MEMORY were set once in Initialize, so
only one pooled connection carried them and the other nine got no busy_timeout,
which drove many "database is locked" errors.

Move the per-connection settings into the DSN so every pooled connection gets a
64 MiB cache (640 MiB worst case over ten connections), synchronous OFF, a
5 s busy_timeout and WAL. Drop those pragmas from Initialize; DISTINCT temp
B-trees now spill to disk. Add process-wide soft (1 GiB) and hard (1.5 GiB)
heap limits; at the hard limit a statement returns SQLITE_NOMEM and the
existing batch paths log and drop, so nothing panics or exits.

Model: opus-4-8
 2026-09-21 15:46:34 +02:00
clawbot 54014c88c8 Run make check inside the Docker build with a pinned lint stage (closes #5) 
REPO_POLICIES.md requires the container build to run the checks; the
Dockerfile only compiled the binary, so script/cibuild checked nothing. A
new lint stage on the golangci-lint v2.7.2 image runs make fmt-check and
make lint, and the build stage waits for it and then runs make test, so
make docker now fails when formatting, lint or a test fails. All three base
images are pinned by digest with their versions unchanged. The final image
and how it starts are unchanged.

make docker is now the gate to use on hosts where the installed linter is
older than the Go toolchain and make lint cannot run directly.

Model: opus-4-8 (implementation, review); fable-5-1 (summary)
 2026-09-21 09:35:16 +02:00
clawbot b0cd884019 Skip live network feed test under -short for deterministic make check (closes #2) 
TestRouteWatchLiveFeed streams the live RIPE RIS feed for a few seconds, so
the default test run depended on the network and tripped the race detector.
It now skips under testing.Short(), and script/test passes -short in both
go test lines, so make check runs offline and gives the same result every
time. The README says how to run the live test on demand. No non-test code
changed.

The race itself is not fixed: the test reads the peeringHandler field of
RouteWatch while Run is still setting it. Nothing outside the test reads
that field concurrently.

Model: opus-4-8 (implementation, review); fable-5-1 (summary)
 2026-09-21 09:05:33 +02:00
sneak ddf0b2fa99 Refresh vendored REPO_POLICIES.md 2026-07-07 01:53:25 +02:00
sneak 583c16a65c Adopt scripts-to-rule-them-all: script/ entrypoints, Makefile shims 2026-07-07 01:53:25 +02:00
sneak 0747165ab1 Add standard Workflow section to TODO.md 2026-07-06 21:06:41 +02:00
sneak b144e1515b Add TODO.md 2026-07-06 20:35:48 +02:00
sneak 68cc06690b Use TRUNCATE mode for WAL checkpoints 
PASSIVE checkpoints may not fully checkpoint when writers are active.
TRUNCATE mode blocks writers briefly but ensures complete checkpointing,
keeping the WAL small for consistent read performance under heavy load.
 2026-01-01 06:49:58 -08:00
sneak 4f62b280c5 Reduce WAL checkpoint interval from 30s to 5s 
Under heavy write load, 30 seconds is too long between checkpoints,
causing the WAL to grow and slow down read queries. More aggressive
checkpointing keeps the WAL small and maintains read performance.
 2026-01-01 06:48:58 -08:00
sneak f1d7c21478 Run WAL checkpoint on startup with logging 
Explicitly checkpoint the WAL on database initialization to consolidate
any large WAL file from previous runs. Log the checkpoint results to
help diagnose issues.
 2026-01-01 06:38:15 -08:00
sneak a163449a28 Improve request logging and make health check lightweight 
- Log slow requests (>1s) at WARNING level with slow=true flag
- Log request timeouts at WARNING level in TimeoutMiddleware
- Replace heavy GetStatsContext with lightweight Ping in health check
- Add Ping method to database interface (SELECT 1)
 2026-01-01 06:06:20 -08:00
sneak 8f524485f7 Add periodic WAL checkpointing to fix slow queries 
The WAL file was growing to 700MB+ which caused COUNT(*) queries to
timeout. Reads must scan the WAL to find current page versions, and
a large WAL makes this slow.

Add Checkpoint method to database interface and run PASSIVE checkpoints
every 30 seconds via the DBMaintainer. This keeps the WAL small and
maintains fast read performance under heavy write load.
 2026-01-01 05:42:03 -08:00
sneak c6fa2b0fbd Fix container to run app as routewatch user 
Use runuser to drop privileges and execute the app as the routewatch
user (uid 1000). Fix data directory permissions at runtime since host
mounts may have incorrect ownership.
 2025-12-31 16:17:59 -08:00
sneak f788a0dbf9 set state dir properly in container 2025-12-31 16:08:17 -08:00
sneak aebdd1b23e Add oldest and newest route timestamps to status page 
Display oldest and newest route timestamps in the Routing Table card
on the /status page. Timestamps are shown as relative times (e.g.,
"5m ago", "2h 30m ago").

Changes:
- Add OldestRoute and NewestRoute fields to database.Stats
- Query MIN/MAX of last_updated across live_routes_v4 and v6 tables
- Include timestamps in both /status.json and /api/v1/stats responses
- Add formatRelativeTime JavaScript function for display
 2025-12-31 15:47:57 -08:00
sneak 8fc10ae98d Fix NULL handling in GetWHOISStats query 
When the asns table is empty, SUM() returns NULL which cannot be
scanned into an int. Wrap SUM expressions in COALESCE to return 0
instead of NULL.
 2025-12-31 15:17:09 -08:00
sneak d27536812f Remove heading from status page 2025-12-31 15:11:03 -08:00
sneak 58b5333c6c Fix navbar and simplify templates 
- Fix nested anchor tags in navbar (invalid HTML)
- Hardcode app name and author in templates
- Remove hero section from index page
 2025-12-31 15:10:24 -08:00
sneak 4284e923a6 Add navbar and home page with search functionality 
- Create new home page (/) with overview stats, ASN lookup,
  AS name search, and IP address lookup with JSON display
- Add responsive navbar to all pages with app branding
- Navbar shows "routewatch by @sneak" with link to author
- Status page accessible via navbar link
- Remove redirect from / to /status, serve home page instead
 2025-12-31 14:56:02 -08:00
sneak 45810e3fc8 Fix prefix URL routing for encoded CIDR notation 
Change route from wildcard /prefix/* to explicit /prefix/{prefix}/{len}
to properly handle URL-encoded IPv6 addresses with CIDR notation.

- Separate prefix and length into individual path parameters
- Add prefixURL template function for generating correct links
- Remove url.QueryUnescape from handlers (chi handles decoding)
 2026-01-01 05:37:37 +07:00
sneak 27909e021f ci 2025-12-31 06:14:44 +01:00
sneak c116b035bd Add status page enhancements with new metrics and footer 
- Add GC statistics (run count, total/last pause, heap usage)
- Add BGP peer count tracking from RIS Live OPEN/NOTIFICATION messages
- Add route churn rate metric (announcements + withdrawals per second)
- Add announcement and withdrawal counters
- Add footer with attribution, license, and git revision
- Embed git revision at build time via ldflags
- Update HTML template to display all new metrics
 2025-12-30 14:50:54 +07:00
sneak 1115954827 Fix prefix URL routing to handle CIDR notation with slashes 
- Use wildcard route pattern for /prefix/* endpoints
- Extract prefix parameter using chi.URLParam(r, "*")
- Fixes 400 error when accessing /prefix/x.x.x.x/32 directly
 2025-12-30 14:41:57 +07:00
sneak 9043cf9bc0 Add connection duration and reconnect count to status page 
- Track reconnection count in metrics tracker
- Display connection duration under Stream Statistics
- Display reconnect count since app startup
- Update both JSON API and HTML status page
 2025-12-30 14:33:37 +07:00
sneak 3a9ec98d5c Add structured HTTP request logging and increase timeouts 
- Replace chi's Logger middleware with structured slog-based logging
- Log request start (debug) and completion (info/warn/error by status)
- Include method, path, status, duration_ms, remote_addr in logs
- Increase request timeout from 8s to 30s for slow queries
- Add read/write/idle timeouts to HTTP server config
- Better server startup logging to confirm listening state
 2025-12-30 13:37:54 +07:00
sneak 0ae89c33db Fix Dockerfile: vendor dependencies after copying source 2025-12-30 13:16:35 +07:00
sneak 8e79b8c074 Add Dockerfile with multi-stage build and source archive 
- Builder stage: vendor dependencies, build binary, create source archive
- Source archive (.tar.zst) includes all code and vendored dependencies
- Runtime stage: minimal Debian image with binary and source archive
- Health check via curl to /.well-known/healthcheck.json
- Runs as non-root user (routewatch:1000)
 2025-12-29 16:07:11 +07:00
sneak 5d7358fce6 Clean up auto_vacuum comment for fresh database deployment 2025-12-29 16:02:27 +07:00
sneak d7e6f46320 Switch to incremental vacuum for non-blocking space reclamation 
- Use PRAGMA incremental_vacuum instead of full VACUUM
- Frees ~1000 pages (~4MB) per run without blocking writes
- Run every 10 minutes instead of 6 hours since it's lightweight
- Set auto_vacuum=INCREMENTAL pragma for new databases
- Remove blocking VACUUM on startup
 2025-12-29 16:00:33 +07:00
sneak da6d605e4d Add production hardening: health check, streamer panic fix, db maintenance 
- Add health check endpoint at /.well-known/healthcheck.json that
  verifies database and RIS Live connectivity, returns 200/503

- Fix panic in streamer when encountering unknown RIS message types
  by logging a warning and continuing instead of crashing

- Add DBMaintainer for periodic database maintenance:
  - VACUUM every 6 hours to reclaim space
  - ANALYZE every hour to update query statistics
  - Graceful shutdown support

- Add Vacuum() and Analyze() methods to database interface
 2025-12-29 15:55:54 +07:00
sneak d2041a5a55 Add WHOIS stats to status page with adaptive fetcher improvements 
- Add WHOIS Fetcher card showing fresh/stale/never-fetched ASN counts
- Display hourly success/error counts and current fetch interval
- Increase max WHOIS rate to 1/sec (down from 10 sec minimum)
- Select random stale ASN instead of oldest for better distribution
- Add index on whois_updated_at for query performance
- Track success/error timestamps for hourly stats
- Add GetWHOISStats database method for freshness statistics
 2025-12-27 16:20:09 +07:00
sneak f8b7d3b773 Unify IP lookup response structure and add PTR lookups 
- Always return consistent JSON structure with query and results array
- Add PTR field to IPInfo for reverse DNS records
- Support comma-separated IPs and hostnames in single query
- Do PTR lookup for all IPs (direct, resolved from hostname, or listed)
- Remove trailing dots from PTR records
 2025-12-27 15:56:10 +07:00
sneak cb75409647 Add hostname resolution support to IP lookup endpoint 
- Accept hostnames in addition to IP addresses for /ip endpoints
- Resolve A and AAAA records for hostnames
- Return list of results with info for each resolved IP
- Include hostname in response when resolving hostnames
- Report per-IP errors while still returning successful lookups
 2025-12-27 15:53:14 +07:00
sneak 8eaf4e5f4b Add adaptive rate limiting to ASN WHOIS fetcher 
- Reduce base interval from 60s to 15s for faster initial fetching
- Add exponential backoff on failure (up to 5 minute max interval)
- Decrease interval on success (down to 10 second minimum)
- Add mutex to prevent concurrent WHOIS fetches
- Track consecutive failures for backoff calculation
 2025-12-27 15:51:06 +07:00
sneak 3b159454eb Add IP information API with background WHOIS fetcher 
- Add /ip and /ip/{addr} JSON endpoints returning comprehensive IP info
- Include ASN, netblock, country code, org name, abuse contact, RIR data
- Extend ASN schema with WHOIS fields (country, org, abuse contact, etc)
- Create background WHOIS fetcher for rate-limited ASN info updates
- Store raw WHOIS responses for debugging and data preservation
- Queue on-demand WHOIS lookups when stale data is requested
- Refactor handleIPInfo to serve all IP endpoints consistently
 2025-12-27 15:47:35 +07:00
sneak 7e4dc528bd Display wire bytes on status page instead of decompressed bytes 
The decompressed stream size is an implementation detail; users care
about actual network bandwidth consumption.
 2025-12-27 12:59:50 +07:00
sneak ab392d874c Track wire bytes separately from decompressed stream bytes 
The stream stats were showing decompressed data sizes, not actual wire
bandwidth. This change adds wire byte tracking by disabling automatic
gzip decompression in the HTTP client and wrapping the response body
with a counting reader before decompression. Both wire (compressed) and
decompressed bytes are now tracked and exposed in the API responses.
 2025-12-27 12:56:57 +07:00
sneak 95bbb655ab Add godoc documentation and README with code structure 
Add comprehensive godoc comments to all exported types, functions,
and constants throughout the codebase. Create README.md documenting
the project architecture, execution flow, database schema, and
component relationships.
 2025-12-27 12:30:46 +07:00
sneak 23dcdd800b Improve godoc documentation for PeeringHandler 
Enhance documentation comments for constants, types, and exported methods
in peeringhandler.go to follow Go documentation conventions. The improved
comments provide more context about the purpose and behavior of each item.
 2025-12-27 12:26:07 +07:00
sneak c292fef0ac Add comprehensive godoc documentation to handler.go 
Expand documentation comments for SimpleHandler type and its methods
to better explain their purpose, parameters, and behavior.
 2025-12-27 12:24:36 +07:00
sneak e1d0ab5ea6 Add detailed godoc documentation to CLIEntry function 
Expand the documentation comment for CLIEntry to provide more context
about what the function does, including its use of the fx dependency
injection framework, signal handling, and blocking behavior.
 2025-12-27 12:24:22 +07:00
sneak 8323a95be9 latest 2025-12-27 12:19:20 +07:00
sneak 2f96141e48 Fix IPv6 prefix length links to use separate /prefixlength6/<len> route 
The prefix length links for IPv6 prefixes were incorrectly pointing to
/prefixlength/<len> which would show IPv4 prefixes. Added a new route
/prefixlength6/<len> specifically for IPv6 prefixes and updated the
template to use the correct URL based on whether displaying IPv4 or IPv6
prefix distributions.

Also refactored handlePrefixLength to explicitly handle only IPv4 prefixes
and created handlePrefixLength6 for IPv6 prefixes, removing the ambiguous
auto-detection based on mask length value.
 2025-08-09 11:37:14 +02:00
sneak 1ec0b3e7ca Change stats fetch interval from 500ms to 2 seconds 
Reduces the frequency of stats API calls from twice per second
to once every 2 seconds, reducing server load.
 2025-07-29 04:22:06 +02:00
sneak 037bbfb813 Reduce slow query threshold from 50ms to 25ms 
This will help identify performance issues earlier by logging
any database query that takes longer than 25 milliseconds.
 2025-07-29 04:20:43 +02:00
sneak 1fded42651 Quadruple all HTTP timeouts to prevent timeout errors 
- HTTP request timeout: 2s -> 8s
- Stats collection context timeout: 1s -> 4s
- HTTP read header timeout: 10s -> 40s

This should prevent timeout errors when the database is under load
or when complex queries take longer than expected (slow query
threshold is 50ms).
 2025-07-29 04:18:07 +02:00
sneak 3338e92785 Add JSON validation middleware to ensure valid API responses 
- Created JSONValidationMiddleware that validates all JSON responses
- Ensures that even on timeout or internal errors, a valid JSON error response is returned
- Applied to all API endpoints including /status.json
- Prevents client-side JSON parse errors when server encounters issues
 2025-07-29 04:13:01 +02:00
sneak 7aec01c499 Add AS peers display to AS detail page 
- Added GetASPeers method to database to fetch all peering relationships
- Updated AS detail handler to fetch and pass peers to template
- Added peers section to AS detail page showing all peer ASNs with their info
- Added peer count to the info cards at the top of the page
- Shows handle, description, and first/last seen dates for each peer
 2025-07-29 03:58:09 +02:00
sneak deeedae180 Fix template references to renamed ASN fields 
Updated templates to use the new field names after renaming:
- ASN.Number -> ASN.ASN in as_detail.html
- Fixed references to ASN field in prefix_detail.html for ASNInfo and ASPathEntry structs
 2025-07-29 03:37:07 +02:00
sneak d3966f2320 Fix SQL query to use renamed asn column 
Fixed remaining references to a.number that should be a.asn after
the column rename in the ASNs table.
 2025-07-29 02:52:47 +02:00
sneak 23127b86e9 Add queue high water marks to handler statistics 
- Track the maximum queue length seen for each handler
- Display high water marks on the status page with percentage
- Helps identify which handlers are experiencing queue pressure
 2025-07-29 02:46:53 +02:00
sneak 2cfca78464 Reduce peering processing interval from 2 minutes to 30 seconds 
The 2 minute interval was causing a noticeable delay before peerings
appeared in the database. Reducing to 30 seconds provides a better
user experience while still maintaining efficient batch processing.
 2025-07-28 23:05:58 +02:00
sneak c9da20e630 Major schema refactoring: simplify ASN and prefix tracking 
- Remove UUID primary keys from ASNs table, use ASN number as primary key
- Update announcements table to reference ASN numbers directly
- Rename asns.number column to asns.asn for consistency
- Add prefix tracking to PrefixHandler to populate prefixes_v4/v6 tables
- Add UpdatePrefixesBatch method for efficient batch updates
- Update all database methods and models to use new schema
- Fix all references in code to use ASN field instead of Number
- Update test mocks to match new interfaces
 2025-07-28 22:58:55 +02:00
sneak a165ecf759 Fix prefix stats by counting from live routes tables 
The prefixes_v4 and prefixes_v6 tables were never being populated
because GetOrCreatePrefix was not being called anywhere. Since we
already track all prefixes in live_routes_v4 and live_routes_v6,
update stats queries to count distinct prefixes from those tables.
 2025-07-28 22:44:44 +02:00
sneak 725d04ffa8 Split prefixes table into prefixes_v4 and prefixes_v6 
- Create separate tables for IPv4 and IPv6 prefixes in schema.sql
- Update indexes for new prefix tables
- Update getOrCreatePrefix to use appropriate table based on IP version
- Update GetStatsContext to count prefixes from both tables
- Remove ip_version column since it's implicit in the table name
 2025-07-28 22:41:42 +02:00
sneak fc32090483 Fix JavaScript UI and complete database table migration 
- Update status page JavaScript to reset all fields to '-' on error
- Fix status page to not show 'Connected' when API returns error
- Update remaining database methods to use new live_routes_v4/v6 tables
- Fix GetStatsContext to count routes from both IPv4 and IPv6 tables
- Fix UpsertLiveRoute to insert into correct table based on IP version
- Fix DeleteLiveRoute to determine table from prefix IP version
 2025-07-28 22:39:01 +02:00
sneak 3673264552 Separate IPv4 and IPv6 routes into different tables 
- Create live_routes_v4 and live_routes_v6 tables
- Update all database methods to use appropriate table
- Add IP version detection in database queries
- Remove filtering by ip_version column for better performance
- Fix route count queries that were timing out
- Update PrefixHandler to include IP version in deletions
 2025-07-28 22:29:15 +02:00
sneak 8e12c07396 Implement queue backpressure with gradual message dropping 
- Add gradual message dropping based on queue utilization
- Start dropping messages at 50% queue capacity
- Drop rate increases linearly from 0% at 50% to 100% at full
- Uses random drops to maintain fair distribution
- Helps prevent queue overflow under high load
 2025-07-28 22:17:00 +02:00
sneak b6ad50f23f Add warnings about schema changes and remove ad-hoc index creation 
- Remove ad-hoc index creation from database.go Initialize method
- Add clear comments to both database.go and schema.sql warning that
  ALL schema changes must be made in schema.sql only
- We do not support migrations, schema changes outside schema.sql are forbidden
 2025-07-28 22:09:19 +02:00
sneak c35b76deb8 Optimize AS detail queries and increase PrefixHandler batch size 
- Increase PrefixHandler batch size from 20k to 25k (25% increase)
- Add missing index on origin_asn for live_routes table
- This index significantly speeds up AS detail page queries
- Add code to create missing indexes on existing databases
 2025-07-28 22:07:27 +02:00
sneak 6d46bbad5b Fix nil pointer dereference in GetPrefixDistributionContext 
- Use separate variables for IPv4 and IPv6 query results
- Add nil checks before closing rows to prevent panic
- Prevents crash when database queries timeout or fail
 2025-07-28 22:04:22 +02:00
sneak 9518519208 Fix prefix links on prefix length page with URL encoding 
- Add urlEncode template function to properly encode prefix URLs
- Move prefix_length.html to embedded templates with function map
- Prevents broken links for prefixes containing slashes
 2025-07-28 22:00:27 +02:00
sneak 7d39bd18bc Fix concurrent map write panic in timeout middleware 
- Add thread-safe header wrapper in timeoutWriter
- Check context cancellation before writing responses in handlers
- Protect header access after timeout with mutex
- Prevents race condition when requests timeout while handlers are still running
 2025-07-28 21:54:58 +02:00
sneak e0a4c8642e Add context cancellation support to database operations 
- Add context-aware versions of all read operations in the database
- Update handlers to use context from HTTP requests
- Allows database queries to be cancelled when HTTP requests timeout
- Prevents database operations from continuing after client disconnects
 2025-07-28 19:27:55 +02:00
sneak 0196251906 Fix race condition crash in timeout middleware 
- Remove duplicate http.Error call when context times out
- The timeout middleware already handles writing the response
- Prevents "concurrent write to websocket connection" panic
 2025-07-28 19:07:30 +02:00
sneak 62ed5e08aa Improve prefix count link styling on status page 
- Add dashed underline to prefix count links to indicate they are clickable
- Change to solid blue underline on hover for better UX
- Remove inline styles and use CSS classes instead
 2025-07-28 19:05:45 +02:00
sneak 5fb3fc0381 Fix prefix length page to show unique prefixes only 
- Change GetRandomPrefixesByLength to return unique prefixes instead of all routes
- Use CTE to first select random unique prefixes, then join to get their latest route info
- This ensures each prefix appears only once in the list
 2025-07-28 19:04:19 +02:00
sneak 9a63553f8d Add index to optimize COUNT(DISTINCT prefix) queries 
- Add compound index on (ip_version, mask_length, prefix) to speed up prefix distribution queries
- This index will significantly improve performance of COUNT(DISTINCT prefix) operations
- Note: Existing databases will need to manually create this index or recreate the database
 2025-07-28 19:01:45 +02:00
sneak ba13c76c53 Fix prefix distribution bug and add prefix length pages 
- Fix GetPrefixDistribution to count unique prefixes using COUNT(DISTINCT prefix) instead of COUNT(*)
- Add /prefixlength/<length> route showing random sample of 500 prefixes
- Make prefix counts on status page clickable links to prefix length pages
- Add GetRandomPrefixesByLength database method
- Create prefix_length.html template with sortable table
- Show prefix age and origin AS with descriptions
 2025-07-28 18:42:38 +02:00
sneak 1dcde74a90 Update AS path display to show handles with clickable links 
- Change AS path from descriptions to handles (short names)
- Make each AS in the path a clickable link to /as/<asn>
- Add font-weight to AS links in path for better visibility
- Prevent word wrapping on all table columns except AS path
- Remove unused maxASDescriptionLength constant
 2025-07-28 18:31:35 +02:00
sneak 81267431f7 Increase batch sizes to improve write throughput 
- Increase prefix batch size from 5K to 20K
- Increase ASN batch size from 10K to 30K
- Add comments warning not to reduce batch timeouts
- Add comments warning not to increase queue sizes above 100K
- Maintains existing batch timeouts for efficiency
 2025-07-28 18:27:42 +02:00
sneak dc3ceb8d94 Show AS descriptions in AS path on prefix detail page 
- Display AS descriptions alongside AS numbers in format: Description (ASN)
- Truncate descriptions longer than 20 characters with ellipsis
- Increase container max width to 1600px for better display
- Enable word wrapping for AS path cells to handle long paths
- Update mobile responsive styles for AS path display
 2025-07-28 18:25:26 +02:00
sneak a78e5c6e92 Add log.txt to .gitignore 2025-07-28 18:13:07 +02:00
sneak 9ef2a22db3 Remove SQLite pragmas that set default values 
- Remove page_size, wal_autocheckpoint, locking_mode, mmap_size
- Keep only pragmas that change behavior from defaults
- Increase cache size to 3GB (upper limit for 2.4GB database)
 2025-07-28 18:12:25 +02:00
sneak 05805b8847 Optimize SQLite settings for better balance 
- Reduce cache size from 8GB to 512MB (still plenty for 2.4GB DB)
- Reduce mmap_size from 10GB to 256MB (reasonable default)
- Use default page size (4KB) instead of 8KB
- Use default WAL checkpoint interval (1000 pages)
- Remove redundant pragmas (threads, cache_spill, read_uncommitted)
- Clean up connection string to only use _txlock parameter
- Keep synchronous=OFF for performance (since we have mutex protection)
 2025-07-28 18:06:31 +02:00
sneak ddb3cfa4f0 Add mutex to serialize database access 
- Add internal mutex to Database struct with lock/unlock wrappers
- Add debug logging for lock acquisition and release with timing
- Wrap all write operations with database mutex
- Use _txlock=immediate in SQLite connection string

This works around apparent issues with SQLite's internal locking
not properly respecting busy_timeout in production environment.
 2025-07-28 17:56:26 +02:00
sneak 3ef60459b2 Fix SQLite transaction deadlocks with immediate mode 
- Add _txlock=immediate to SQLite connection string
- This prevents deadlocks by acquiring write locks at transaction start
- Multiple concurrent writers now queue properly instead of failing instantly
- Resolves 'database table is locked' errors in production
 2025-07-28 17:26:42 +02:00
sneak 40d7f0185b Optimize database batch operations with prepared statements 
- Add prepared statements to all batch operations for better performance
- Fix database lock contention by properly batching operations
- Update SQLite settings for extreme performance (8GB cache, sync OFF)
- Add proper error handling for statement closing
- Update tests to properly track batch operations
 2025-07-28 17:21:40 +02:00
sneak b9b0792df9 Fix shutdown handling and optimize SQLite settings 
- Fix Ctrl-C shutdown by using fx.Shutdowner instead of just canceling context
- Pass context from fx lifecycle to rw.Run() for proper cancellation
- Adjust WAL settings: checkpoint at 50MB, max size 100MB
- Reduce busy timeout from 30s to 2s to fail fast on lock contention

This should fix the issue where Ctrl-C doesn't cause shutdown and improve
database responsiveness under heavy load.
 2025-07-28 16:52:52 +02:00
sneak 21921a170c Optimize database performance to fix slow queries 
- Add VACUUM on startup to defragment database
- Increase cache size from 256MB to 2GB for better performance
- Increase mmap_size from 256MB to 512MB
- Add PRAGMA analysis_limit=0 to disable automatic ANALYZE
- Remove PRAGMA optimize which could trigger slow ANALYZE

These changes should dramatically improve query performance and prevent
the 5+ second query times seen in production.
 2025-07-28 16:47:59 +02:00
sneakandClaude 78d6e17c76 Add debug logging and optimize SQLite performance 
- Add debug logging for goroutines and memory usage (enabled via DEBUG=routewatch)
- Increase SQLite connection pool from 1 to 10 connections for better concurrency
- Optimize SQLite pragmas for balanced performance and safety
- Add proper shutdown handling for peering handler
- Define constants to avoid magic numbers in code

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-07-28 15:45:06 +02:00
sneak 9b649c98c9 Fix AS detail view and add prefix sorting 
- Fix GetASDetails to properly handle timestamp from MAX(last_updated)
- Parse timestamp string from SQLite aggregate function result
- Add natural sorting of prefixes by IP address in AS detail view
- Sort IPv4 and IPv6 prefixes separately by network address
- Remove SQL ORDER BY since we're sorting in Go
- This fixes the issue where AS detail pages showed no prefixes
 2025-07-28 04:42:10 +02:00
sneak 48db8b9edf Fix AS detail view to show unique prefixes 
- Update GetASDetails query to GROUP BY prefix instead of using DISTINCT
- Use MAX(last_updated) to get the most recent update time for each prefix
- This prevents duplicate prefixes from appearing when announced by multiple peers
- Maintains the same prefix count and ordering
 2025-07-28 04:36:22 +02:00
sneak df31cf880a Fix prefix URL routing by using URL encoding 
- Replace slash-to-dash conversion with proper URL encoding
- Update handlePrefixDetail and handlePrefixDetailJSON to URL decode prefix parameter
- Update handleIPRedirect to URL encode the prefix in the redirect
- Add urlEncode template function for use in templates
- Update AS detail template to URL encode prefix links
- This properly handles the slash in CIDR notation (e.g., /prefix/192.168.1.0%2F24)
 2025-07-28 04:34:34 +02:00
sneak af9ff258b1 Add /ip/<ip> route that redirects to prefix detail page 
- Implement handleIPRedirect handler that looks up the prefix containing an IP
- Add /ip/{ip} route to routes.go
- Reuse existing GetASInfoForIP database method which returns prefix info
- Redirect to /prefix/<prefix> page with HTTP 303 See Other status
- Handle invalid IPs (400) and IPs with no route (404)
 2025-07-28 04:31:22 +02:00
sneak aeeb5e7d7d Implement AS and prefix detail pages 
- Implement handleASDetail() and handlePrefixDetail() HTML handlers
- Create AS detail HTML template with prefix listings
- Create prefix detail HTML template with route information
- Add timeSince template function for human-readable durations
- Update templates.go to include new templates
- Server-side rendered pages as requested (no client-side API calls)
 2025-07-28 04:26:20 +02:00
sneak 27ae80ea2e Refactor server package: split handlers and routes into separate files 
- Move all handler functions to handlers.go
- Move setupRoutes to routes.go
- Clean up server.go to only contain core server logic
- Add missing GetASDetails and GetPrefixDetails to mockStore for tests
- Fix linter errors (magic numbers, unused parameters, blank lines)
 2025-07-28 04:00:12 +02:00
sneak 2fc24bb937 Add route age information to IP lookup API 
- Add last_updated timestamp and age fields to ASInfo
- Include route's last_updated time from live_routes table
- Calculate and display age as human-readable duration
- Update both IPv4 and IPv6 queries to fetch timestamp
- Fix error handling to return 400 for invalid IPs
 2025-07-28 03:44:19 +02:00
sneak 691710bc7c Add /api/v1/ip/<ip> endpoint for IP to AS lookups 
- Add handleIPLookup handler that uses GetASInfoForIP
- Create writeJSONError and writeJSONSuccess helper functions
- Refactor all JSON error responses to use the helpers
- Add GetASInfoForIP to Store interface
- Add mock implementation for tests
- Fix all linter warnings
 2025-07-28 03:31:53 +02:00
sneak afb916036c Fix handler processing time display for sub-millisecond values 
- Add formatProcessingTime function to display microseconds for values < 1ms
- Show 0 µs for times < 0.001ms, X.X µs for times < 0.01ms
- Show X.XXX ms for times < 1ms, X.XX ms for times >= 1ms
- Apply formatting to both average and min/max time displays
 2025-07-28 03:26:16 +02:00
sneak 13047b5cb9 Add IPv4 range optimization for IP to AS lookups 
- Add v4_ip_start and v4_ip_end columns to live_routes table
- Calculate IPv4 CIDR ranges as 32-bit integers for fast lookups
- Update PrefixHandler to populate IPv4 range fields
- Add GetASInfoForIP method with optimized IPv4 queries
- Add comprehensive tests for IP conversion functions
 2025-07-28 03:23:25 +02:00
sneak ae89468a1b Remove routing table and snapshotter packages, update status page 
- Remove routingtable package entirely as database handles all routing data
- Remove snapshotter package as database contains all information
- Rename 'Connection Status' box to 'RouteWatch' and add Go version, goroutines, memory usage
- Move IPv4/IPv6 prefix counts from Database Statistics to Routing Table box
- Add Peers count to Database Statistics box
- Add go-humanize dependency for memory formatting
- Update server to include new metrics in API responses
 2025-07-28 03:11:36 +02:00
sneak d929f24f80 Remove RoutingTableHandler and snapshotter, use database for route stats 
- Remove RoutingTableHandler as PrefixHandler maintains live_routes table
- Update server to get route counts from database instead of in-memory routing table
- Add GetLiveRouteCounts method to database for IPv4/IPv6 route counts
- Use metrics tracker in PrefixHandler for route update rates
- Remove snapshotter entirely as database contains all information
- Update tests to work without routing table
 2025-07-28 03:02:44 +02:00
sneak cb1f4d9052 Add route update metrics tracking to PrefixHandler 
- Add RecordIPv4Update and RecordIPv6Update to metrics package
- Add SetMetricsTracker method to PrefixHandler
- Track IPv4/IPv6 route updates when processing announcements
- Add GetMetricsTracker method to Streamer to expose metrics
 2025-07-28 02:55:27 +02:00
sneak bc640b0b37 Reduce all handler queue sizes to 100,000 2025-07-28 02:50:05 +02:00
sneak 7d814c9d2d Optimize SQLite and PrefixHandler for better performance 
- Increase PrefixHandler queue size to 500k and batch size to 25k
- Set SQLite PRAGMA synchronous=OFF for faster writes (trades durability)
- Increase SQLite cache to 1GB and mmap to 512MB
- Increase WAL checkpoint interval to 10000 pages
- Set page size to 8KB for better performance
- Increase busy timeout to 30 seconds
- Keep single connection to avoid SQLite locking issues
 2025-07-28 02:40:17 +02:00
sneak 54bb0ba1cb Simplify peerings table to store AS numbers directly 
- Rename asn_peerings table to peerings
- Change columns from from_asn_id/to_asn_id to as_a/as_b (integers)
- Remove foreign key constraints to asns table
- Update RecordPeering to use AS numbers directly
- Add validation in RecordPeering to ensure:
  - Both AS numbers are > 0
  - AS numbers are different
  - as_a is always lower than as_b (normalized)
- Update PeeringHandler to no longer need ASN cache
- Simplify the code by removing unnecessary ASN lookups
 2025-07-28 02:36:15 +02:00
sneak 1157003db7 Refactor database handlers and optimize PeeringHandler 
- Create PeeringHandler for asn_peerings table maintenance
- Rename DBHandler to ASHandler (now only handles asns table)
- Move prefixes table maintenance to PrefixHandler
- Optimize PeeringHandler with in-memory AS path tracking:
  - Stores AS paths in memory with timestamps
  - Processes peerings in batch every 2 minutes
  - Prunes old paths (>30 minutes) every 5 minutes
  - Normalizes peerings with lower AS number first
- Each handler now has a single responsibility:
  - ASHandler: asns table
  - PeerHandler: bgp_peers table
  - PrefixHandler: prefixes and live_routes tables
  - PeeringHandler: asn_peerings table
 2025-07-28 02:31:04 +02:00
sneak eaa11b5f8d Move prefixes table maintenance from DBHandler to PrefixHandler 
- DBHandler now only maintains asns and asn_peerings tables
- PrefixHandler maintains both prefixes and live_routes tables
- This consolidates all prefix-related operations in one handler
 2025-07-28 02:16:12 +02:00
sneak 8b43882526 Increase batch sizes to 10000 and queue sizes to 200000, reduce timeout to 2s 2025-07-28 02:11:05 +02:00
sneak eda90d96a9 Remove debug logging for withdrawals without origin ASN 2025-07-28 02:07:33 +02:00
sneak 3c46087976 Add live routing table with CIDR mask length tracking 
- Added new live_routes table with mask_length column for tracking CIDR prefix lengths
- Updated PrefixHandler to maintain live routing table with additions and deletions
- Added route expiration functionality (5 minute timeout) to in-memory routing table
- Added prefix distribution stats showing count of prefixes by mask length
- Added IPv4/IPv6 prefix distribution cards to status page
- Updated database interface with UpsertLiveRoute, DeleteLiveRoute, and GetPrefixDistribution
- Set all handler queue depths to 50000 for consistency
- Doubled DBHandler batch size to 32000 for better throughput
- Fixed withdrawal handling to delete routes when origin ASN is available
 2025-07-28 01:51:42 +02:00
sneak cea7c3dfd3 Rename handlers and add PrefixHandler for database routing table 
- Renamed BatchedDatabaseHandler to DBHandler
- Renamed BatchedPeerHandler to PeerHandler
- Quadrupled DBHandler batch size from 4000 to 16000
- Created new PrefixHandler using same batching strategy to maintain routing table in database
- Removed verbose batch flush logging from all handlers
- Updated app.go to use renamed handlers and register PrefixHandler
- Fixed test configuration to enable batched database writes
 2025-07-28 01:37:19 +02:00
sneak 3aef3f9a07 Format logger source location as file.go:linenum 
- Change source location format from separate source and line fields
  to a single source field with format 'file.go:linenum'
- This provides a more concise and standard format for source locations
 2025-07-28 01:16:29 +02:00
sneak 67f6b78aaa Add custom logger with source location tracking and remove verbose database logs 
- Create internal/logger package with Logger wrapper around slog
- Logger automatically adds source file, line number, and function name to all log entries
- Use golang.org/x/term to properly detect if stdout is a terminal
- Replace all slog.Logger usage with logger.Logger throughout the codebase
- Remove verbose logging from database GetStats() method
- Update all constructors and dependencies to use the new logger
 2025-07-28 01:14:51 +02:00
sneak 3f06955214 Increase batch sizes and timeouts for better throughput 
- Increase batch size from 100/50 to 500 for both handlers
- Increase batch timeout from 100-200ms to 5 seconds
- This will reduce database write frequency and improve throughput
  for high-volume BGP update streams
 2025-07-28 01:03:09 +02:00
sneak 155c08d735 Implement batched database operations for improved performance 
- Add BatchedDatabaseHandler that batches prefix, ASN, and peering operations
- Add BatchedPeerHandler that batches peer update operations
- Batch operations are deduped and flushed every 100-200ms or when batch size is reached
- Add EnableBatchedDatabaseWrites config option (enabled by default)
- Properly flush remaining batches on shutdown
- This significantly reduces database write pressure and improves throughput
 2025-07-28 01:01:27 +02:00
83 changed files with 14240 additions and 2835 deletions
Expand all files Collapse all files
.dockerignore
+97
View File
@@ -0,0 +1,97 @@
# .dockerignore does NOT use .gitignore semantics. Docker matches with
# moby/patternmatcher: filepath.Match plus `**`, so `*` does not cross
# `/` and an unprefixed pattern is anchored at the context root. Every
# depth-independent pattern therefore needs `**/`, or `config/.env` and
# `certs/server.key` still ship while this file reads as solved. Only
# genuinely root-anchored entries go unprefixed. Never transplant these
# into .gitignore, where `**/` is wrong.
#
# Matching is case-sensitive, so secrets use character ranges rather
# than an ALL-CAPS twin, which would still miss `Server.Key`.
#
# Extend with this repo's own host-built artifacts, written anchored:
# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
# deletes the package directory from the context.
# .git is sent without its config. Without a VERSION build argument the
# stage that compiles runs `git describe --tags --always` on .git, which
# does not need .git/config; that file can hold a credential, such as a
# password in a remote URL or the token the CI checkout step stores there.
# Each submodule keeps a config with the same exposure in its git directory
# under .git/modules/, nested again for a submodule's own submodules, or in
# its own .git directory when it keeps one.
# KNOWN GAP: a submodule whose name has a `config` segment (`config`,
# `deploy/config`, `config/lib`) loses its whole git directory, because
# `**/.git/modules/**/config` also matches that segment's directory
# under .git/modules/. Go's version stamping then fails the build;
# nothing leaks. Name such a submodule without that segment:
# `git submodule add --name`.
**/.git/config
**/.git/modules/**/config
# Agent scratch: one full checkout of the repo per in-flight agent.
# Anchored because it occurs once where agents run at the repo root.
# KNOWN GAP: a repo running agents in subdirectories still ships
# `services/api/.claude/` and must add its own anchored entry.
.claude
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Re-include a committed template with a negation if the
# build needs one: `!docs/example.env`.
**/*.[eE][nN][vV]
**/.[eE][nN][vV].*
**/.[eE][nN][vV][rR][cC]
# Private keys and the bundles carrying them. Public certificates
# (*.crt, *.cer) are deliberately absent: they are legitimate inputs.
**/*.[pP][eE][mM]
**/*.[kK][eE][yY]
**/*.[pP]12
**/*.[pP][fF][xX]
**/[iI][dD]_[rR][sS][aA]
**/[iI][dD]_[dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
**/[iI][dD]_[eE][dD]25519
**/[iI][dD]_[eE][dD]25519_[sS][kK]
# Dependencies: restored inside the image, never copied in.
**/node_modules
# OS metadata.
**/.DS_Store
**/Thumbs.db
# Editor state: never a build input, and it churns COPY.
**/*.swp
**/*.swo
**/*~
**/*.bak
**/.idea
**/.vscode
**/*.sublime-*
# This repository's host-built artifacts: `make build` writes bin/, `go
# build` in the root the binaries, `make run` log.txt, `make asupdate` its
# .tmp file; test binaries, coverage profiles and source archives.
/bin
/routewatch
/asinfo-gen
/streamdumper
/log.txt
/out
/pkg/asinfo/asdata.json
**/*.tar.zst
**/*.test
**/*.out
**/*.tmp
# Local databases. The image carries a source archive of the whole build
# context, so these would otherwise ship inside it.
**/*.db
**/*.db-journal
**/*.db-wal
# A local Go workspace points at directories outside the build context.
/go.work
/go.work.sum
.editorconfig
+15
View File
@@ -0,0 +1,15 @@
root = true
[*]
indent_style = space
indent_size = 4
end_of_line = lf
charset = utf-8
trim_trailing_whitespace = true
insert_final_newline = true
[Makefile]
indent_style = tab
[*.go]
indent_style = tab
.gitea/workflows/check.yml
+9
View File
@@ -0,0 +1,9 @@
name: check
on: [push]
jobs:
check:
runs-on: ubuntu-latest
steps:
# actions/checkout v4.2.2, 2026-02-22
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
- run: script/cibuild
.gitignore
+63 -23
View File
@@ -1,37 +1,77 @@
# Binaries for programs and plugins
# OS
.DS_Store
Thumbs.db
# Editors
*.swp
*.swo
*~
*.bak
.idea/
.vscode/
*.sublime-*
# Agent scratch (worktrees of this repo, created and destroyed by
# in-flight tooling). Unanchored: .gitignore patterns already match at
# every depth, so no prefix is wanted here. This is not a .dockerignore
# entry and must not be given a `**/` prefix on the way into one.
.claude/
# Node
node_modules/
# Secrets. Unanchored like every entry above, so each matches at every
# depth. Matching is case-sensitive on Linux, so names use character
# ranges rather than a lowercase form that misses `Server.Key`.
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Only the templates `example.env` and `sample.env` are
# re-included below. A repository that commits any other template adds
# its own negation after these lines, for example `!.env.example`.
*.[eE][nN][vV]
.[eE][nN][vV].*
.[eE][nN][vV][rR][cC]
!example.env
!sample.env
# Private keys and the bundles carrying them.
*.[pP][eE][mM]
*.[kK][eE][yY]
*.[pP]12
*.[pP][fF][xX]
[iI][dD]_[rR][sS][aA]
[iI][dD]_[dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
[iI][dD]_[eE][dD]25519
[iI][dD]_[eE][dD]25519_[sS][kK]
# This repository's own entries, kept after the canonical content above.
# Go: logs, test binaries, coverage profiles, and the binaries `make build`
# and `go build` in the repository root write.
*.log
*.out
*.test
*.exe
*.exe~
*.dll
*.so
*.dylib
/bin/
# Test binary, built with `go test -c`
*.test
# Output of the go coverage tool, specifically when used with LiteIDE
*.out
# Dependency directories (remove the comment below to include it)
# vendor/
# Go workspace file
/routewatch
/asinfo-gen
/streamdumper
go.work
go.work.sum
# env file
.env
# Database files
# Local databases, temporary files and debug output.
*.db
*.db-journal
*.db-wal
# Temporary files
*.tmp
# Raw AS data (we only commit the gzipped version)
pkg/asinfo/asdata.json
# Debug output files
log.txt
out
# Raw AS data (only the gzipped version is committed).
pkg/asinfo/asdata.json
.golangci.yml
+87 -80
View File
@@ -1,92 +1,99 @@
version: "2"
# Config schema uses the golangci-lint v2 layout (settings live under
# linters.settings, not top-level linters-settings) so that the
# thresholds below are actually applied by golangci-lint >= v2.
run:
go: "1.24"
tests: false
timeout: 5m
modules-download-mode: readonly
linters:
default: all
enable:
# Additional linters requested
- testifylint # Checks usage of github.com/stretchr/testify
- usetesting # usetesting is an analyzer that detects using os.Setenv instead of t.Setenv since Go 1.17
# - tagliatelle # Disabled: we need snake_case for external API compatibility
- nlreturn # nlreturn checks for a new line before return and branch statements
- nilnil # Checks that there is no simultaneous return of nil error and an invalid value
- nestif # Reports deeply nested if statements
- mnd # An analyzer to detect magic numbers
- lll # Reports long lines
- intrange # intrange is a linter to find places where for loops could make use of an integer range
- gochecknoglobals # Check that no global variables exist
# Default/existing linters that are commonly useful
- govet
- errcheck
- staticcheck
- unused
- ineffassign
- misspell
- revive
- gosec
- unconvert
- unparam
linters-settings:
# Successor to the deprecated gomodguard. Named explicitly, rather than
# left to `default: all`, because it carries the module policy below.
- gomodguard_v2
disable:
# Genuinely incompatible with project patterns
- exhaustruct # Requires all struct fields
- exhaustruct_v5 # Requires all struct fields (successor to exhaustruct)
- godot # Requires comments to end with periods
- wrapcheck # Too verbose for internal packages
- varnamelen # Short names like db, id are idiomatic Go
# Deprecated: the warning is attached to the old name, so it is
# silenced by disabling that name, not by enabling the successor.
- wsl # Deprecated, replaced by wsl_v5
- gomodguard # Deprecated, replaced by gomodguard_v2
settings:
lll:
line-length: 120
nestif:
min-complexity: 4
nlreturn:
block-size: 2
revive:
line-length: 88
funlen:
lines: 80
statements: 50
cyclop:
max-complexity: 15
dupl:
threshold: 100
depguard:
# Test-support code must not be compiled into the shipped binary. A
# test-support package exists to hand a test privileges the program
# itself must never have, so a file that is not a test must not import
# one. Test files, and the files inside a package whose directory name
# ends in `test`, are where that code belongs, and are exempt.
#
# The deny list below is the one part of this file a repository is
# expected to extend, and the only part it may. depguard matches an
# import path against a list of prefixes, so it cannot be told "any path
# whose last segment ends in test"; a repository's own test-support
# packages have to be named here one at a time, by full import path,
# under a module path that differs from repository to repository. Add
# them; change nothing else.
rules:
- name: var-naming
arguments:
- []
- []
- "upperCaseConst=true"
tagliatelle:
case:
rules:
json: snake
yaml: snake
xml: snake
bson: snake
testifylint:
enable-all: true
usetesting: {}
test-support:
list-mode: lax
files:
- "$all"
- "!$test"
- "!**/*test/**"
deny:
- pkg: net/http/httptest
desc: >-
Test-support code belongs in test files and in packages whose
directory name ends in test, not in the shipped binary.
# Only decisions already recorded in the Go package defaults are
# listed here. Every entry matches the module path exactly.
gomodguard_v2:
blocked:
- module: github.com/rs/zerolog
recommendations:
- log/slog
reason: "Structured logging is stdlib log/slog."
# One entry per pre-fork module path, because the later releases
# are separate paths. A prefix match would be shorter but would
# also reach github.com/go-redis/redismock, the test double for
# the successor these entries recommend.
- module: github.com/go-redis/redis
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v7
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v8
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/sergi/go-diff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "No unified diff output; use go-udiff."
- module: github.com/hexops/gotextdiff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "Unmaintained fork; use go-udiff."
issues:
max-issues-per-linter: 0
max-same-issues: 0
exclude-rules:
# Exclude unused parameter warnings for cobra command signatures
- text: "parameter '(args|cmd)' seems to be unused"
linters:
- revive
# Allow ALL_CAPS constant names
- text: "don't use ALL_CAPS in Go names"
linters:
- revive
# Allow snake_case JSON tags for external API compatibility
- path: "internal/types/ris.go"
linters:
- tagliatelle
# Allow snake_case JSON tags for database models
- path: "internal/database/models.go"
linters:
- tagliatelle
# Allow generic package name for types that define data structures
- path: "internal/types/"
text: "avoid meaningless package names"
linters:
- revive
CLAUDE.md → AGENTS.md
View File
Dockerfile
+129
View File
@@ -0,0 +1,129 @@
# Lint phase, built alone by script/lint. The linter is invoked directly
# rather than through `make lint`, which is itself a docker build and
# would recurse into a daemon that does not exist in a build step. The
# image is Debian-based and has Go and gcc, so it can check the cgo
# SQLite package without extra installs.
# golangci/golangci-lint:v2.14.0 (Go 1.27.0), 2026-10-14
FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN golangci-lint run --config .golangci.yml ./...
# Test phase, built alone by script/test. -race needs cgo and so a C
# compiler, which the Debian Go image ships. -short skips the live RIPE RIS
# feed test, so the suite needs no network. The rerun with -v only shows
# details: the build fails however it ends, because the first run failed.
# golang:1.24-bookworm, 2026-09-21
FROM golang@sha256:1a6d4452c65dea36aac2e2d606b01b4a029ec90cc1ae53890540ce6173ea77ac AS test
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go test -short -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -short -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from the lint and test phases; the copies
# are what make BuildKit build them first, so this stage cannot run unless
# lint and test passed.
# golang:1.24-bookworm, 2026-09-21
FROM golang@sha256:1a6d4452c65dea36aac2e2d606b01b4a029ec90cc1ae53890540ce6173ea77ac AS builder
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
# Install build dependencies (zstd for archive, gcc for CGO/sqlite3)
RUN apt-get update && apt-get install -y --no-install-recommends \
zstd \
gcc \
libc6-dev \
&& rm -rf /var/lib/apt/lists/*
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src
# Copy everything
COPY . .
# Vendor dependencies (must be after copying source)
RUN go mod download && go mod vendor
# Build the binary with CGO enabled (required for sqlite3). The version the
# page footer shows is the VERSION build argument when one is given, otherwise
# `git describe --tags --always` of the .git in the build context (git comes
# with this image): the tag on a tagged commit, tag-N-gHASH after one, the
# short commit when no tag is reachable. A context that carries .git and still
# yields no version fails the build. The footer links to the full commit.
ARG VERSION
RUN version="${VERSION:-$(git describe --tags --always || echo unknown)}"; \
if [ -e .git ] && { [ -z "$version" ] || [ "$version" = dev ] || \
[ "$version" = unknown ]; }; then \
echo "no version could be derived although the build context carries .git" >&2; \
exit 1; \
fi; \
CGO_ENABLED=1 GOOS=linux go build -o /routewatch -ldflags "\
-X git.eeqj.de/sneak/routewatch/internal/version.GitRevision=$(git rev-parse --verify HEAD || echo unknown) \
-X git.eeqj.de/sneak/routewatch/internal/version.GitRevisionShort=$version" \
./cmd/routewatch
# Create source archive with vendored dependencies
RUN tar --zstd -cf /routewatch-source.tar.zst \
--exclude='.git' \
--exclude='*.tar.zst' \
.
# Runtime stage
# debian:bookworm-slim, 2026-09-21
FROM debian@sha256:3783cc01769c7b2b1b83a5c5ad96c815348e28ed7da68e2e3687004faa906251
# Install runtime dependencies
# - ca-certificates: for HTTPS connections
# - curl: for health checks
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
curl \
&& rm -rf /var/lib/apt/lists/*
# Create non-root user
RUN useradd -r -u 1000 -m routewatch
RUN mkdir -p /var/lib/berlin.sneak.app.routewatch && chown routewatch:routewatch /var/lib/berlin.sneak.app.routewatch
RUN mkdir /app
WORKDIR /app
# Copy binary and source archive from builder
COPY --from=builder /routewatch /app/routewatch
COPY --from=builder /routewatch-source.tar.zst /app/source/routewatch-source.tar.zst
# Set ownership
RUN chown -R routewatch:routewatch /app
ENV XDG_DATA_HOME=/var/lib
# Cap the Go heap at 1.5 GiB so the runtime collects harder before the
# container's memory limit is reached. setpriv in the entrypoint preserves this
# the way it does XDG_DATA_HOME above.
ENV GOMEMLIMIT=1536MiB
# Cap glibc's malloc arenas. The SQLite C library allocates and frees millions
# of small page-cache chunks from many threads; glibc otherwise creates up to
# eight arenas per core (hundreds on a large host) and keeps each arena's freed
# chunks resident, so process RSS climbs far above SQLite's live heap and never
# comes back down. Two arenas keep that retained memory bounded; database writes
# are already serialized, so the lost allocator concurrency costs nothing here.
ENV MALLOC_ARENA_MAX=2
# Expose HTTP port
EXPOSE 8080
COPY ./entrypoint.sh /entrypoint.sh
# Health check using the health endpoint, on the port PORT names
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD curl -sf "http://localhost:${PORT:-8080}/.well-known/healthcheck.json" || exit 1
ENTRYPOINT ["/bin/bash", "/entrypoint.sh" ]
LICENSE
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Jeffrey Paul <sneak@sneak.berlin>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Makefile
+37 -8
View File
@@ -1,27 +1,56 @@
export DEBUG = routewatch
.PHONY: test fmt lint build clean run asupdate
# Git revision for version embedding
GIT_REVISION := $(shell git rev-parse HEAD 2>/dev/null || echo "unknown")
GIT_REVISION_SHORT := $(shell git describe --tags --always 2>/dev/null || echo "unknown")
VERSION_PKG := git.eeqj.de/sneak/routewatch/internal/version
LDFLAGS := -X $(VERSION_PKG).GitRevision=$(GIT_REVISION) -X $(VERSION_PKG).GitRevisionShort=$(GIT_REVISION_SHORT)
.PHONY: bootstrap setup check test lint fmt fmt-check docker hooks build clean run asupdate
all: test
test: lint
go test -v ./...
# Install all development dependencies.
bootstrap:
@script/bootstrap
# Prepare a fresh clone: bootstrap plus pre-commit hook.
setup:
@script/setup
# Combined pre-commit/CI gate: tests, lint, format check.
check:
@script/check
test:
@script/test
fmt:
go fmt ./...
@script/fmt
# Check if code is formatted (read-only).
fmt-check:
@script/fmt-check
lint:
go vet ./...
golangci-lint run
@script/lint
# Build Docker image.
docker:
@script/docker
# Install pre-commit hook.
hooks:
@script/install-precommit
build:
CGO_ENABLED=1 go build -o bin/routewatch cmd/routewatch/main.go
CGO_ENABLED=1 go build -ldflags "$(LDFLAGS)" -o bin/routewatch cmd/routewatch/main.go
clean:
rm -rf bin/
run: build
./bin/routewatch
DEBUG=routewatch ./bin/routewatch 2>&1 | tee log.txt
asupdate:
@echo "Updating AS info data..."
README.md
+320
View File
@@ -0,0 +1,320 @@
# RouteWatch
RouteWatch is an MIT-licensed Go daemon by @sneak that monitors the BGP routing
table in real time: it streams BGP UPDATE messages from the RIPE RIS Live
service, maintains a live routing table in SQLite, and provides HTTP APIs for
querying routing information.
## Features
- Real-time streaming of BGP updates from RIPE RIS Live
- Maintains live IPv4 and IPv6 routing tables
- Tracks AS peering relationships
- HTTP API for IP-to-AS lookups, prefix details, and AS information
- Automatic reconnection with exponential backoff
- Batched database writes for high performance
- Backpressure handling to prevent memory exhaustion
## Installation
```bash
go build -o routewatch ./cmd/routewatch
```
## Usage
```bash
# Run the daemon (listens on port 8080 by default)
./routewatch
# Set custom port
PORT=3000 ./routewatch
# Enable debug logging
DEBUG=routewatch ./routewatch
```
## HTTP Endpoints
### Web Interface
- `GET /` - Redirects to /status
- `GET /status` - HTML status dashboard
- `GET /status.json` - JSON statistics
- `GET /as/{asn}` - AS detail page (HTML)
- `GET /prefix/{prefix}` - Prefix detail page (HTML)
- `GET /prefixlength/{length}` - IPv4 prefixes by mask length
- `GET /prefixlength6/{length}` - IPv6 prefixes by mask length
- `GET /ip/{ip}` - Redirects to prefix containing the IP
### API v1
- `GET /api/v1/stats` - Detailed statistics with handler metrics
- `GET /api/v1/ip/{ip}` - Look up AS information for an IP address
- `GET /api/v1/as/{asn}` - Get prefixes announced by an AS
- `GET /api/v1/prefix/{prefix}` - Get routes for a specific prefix
## Code Structure
```
routewatch/
├── cmd/
│ ├── routewatch/ # Main daemon entry point
│ ├── asinfo-gen/ # Utility to generate AS info data
│ └── streamdumper/ # Debug utility for raw stream output
├── internal/
│ ├── routewatch/ # Core application logic
│ ├── server/ # HTTP server and handlers
│ ├── database/ # SQLite storage layer
│ ├── streamer/ # RIPE RIS Live client
│ ├── ristypes/ # BGP message data structures
│ ├── logger/ # Structured logging wrapper
│ ├── metrics/ # Performance metrics tracking
│ ├── config/ # Configuration management
│ └── templates/ # HTML templates
└── pkg/
└── asinfo/ # AS information lookup (public API)
```
## Architecture Overview
### Component Relationships
```
┌─────────────────────────────────────────────────────────────────┐
│ RouteWatch │
│ (internal/routewatch/app.go - main orchestrator) │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Streamer │───▶│ Handlers │───▶│ Database │ │
│ │ │ │ │ │ │ │
│ │ RIS Live │ │ - ASHandler │ │ SQLite with │ │
│ │ WebSocket │ │ - PeerHandler│ │ WAL mode │ │
│ │ client │ │ - PrefixHdlr │ │ │ │
│ │ │ │ - PeeringHdlr│ │ Tables: │ │
│ └──────────────┘ └──────────────┘ │ - asns │ │
│ │ - prefixes │ │
│ ┌──────────────┐ ┌──────────────┐ │ - live_routes│ │
│ │ Server │───▶│ Handlers │───▶│ - peerings │ │
│ │ │ │ │ │ - bgp_peers │ │
│ │ Chi router │ │ Status, API │ └──────────────┘ │
│ │ port 8080 │ │ AS, Prefix │ │
│ └──────────────┘ └──────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
```
### Execution Flow
1. **Startup** (`cmd/routewatch/main.go` → `internal/routewatch/cli.go`)
- Uber fx dependency injection initializes all components
- Signal handlers registered for graceful shutdown
2. **Initialization** (`internal/routewatch/app.go`)
- Database created with SQLite schema (WAL mode, 3GB cache)
- Message handlers registered with the streamer
- HTTP server started on configured port
3. **Message Processing Pipeline**
```
RIS Live Stream → JSON Parser → Message Dispatcher → Handler Queues → Batch Writers → SQLite
```
- Streamer connects to `ris-live.ripe.net` via HTTP
- Parses BGP UPDATE messages from JSON stream
- Dispatches to registered handlers based on message type
- Each handler has its own queue with backpressure handling
- Handlers batch writes for efficiency (25K-30K ops, 1-2s timeout)
4. **Handler Details**
- **ASHandler**: Tracks all ASNs seen in AS paths
- **PeerHandler**: Records BGP peer information
- **PrefixHandler**: Maintains live routing table (upserts on announcement, deletes on withdrawal)
- **PeeringHandler**: Extracts AS peering relationships from AS paths
5. **HTTP Request Flow**
```
Request → Chi Router → Middleware (timeout, logging) → Handler → Database Query → Response
```
### Key Design Patterns
- **Batched Writes**: All database operations are batched for performance
- **Backpressure**: Probabilistic message dropping when queues exceed 50% capacity
- **Graceful Shutdown**: 60-second timeout, flushes all pending batches
- **Reconnection**: Any end of the stream other than shutdown, including RIS Live
ending the response normally, marks the feed disconnected and reconnects with
exponential backoff (5s-320s). The backoff resets after a stream that lasted
more than 30s. A stream with no line for 60s is dropped and reconnected.
- **IP Lookup**: the most specific live route is found by looking up the
address's prefix at each mask length, longest first, in the prefix index
(at most 25 lookups for IPv4, 113 for IPv6); prefixes are stored in the
text form Go's `net/netip` prints. Routes shorter than /8 for IPv4 or /16
for IPv6, such as a default route, are not used.
### Database Schema
```sql
-- Core tables
asns(id, number, handle, description, first_seen, last_seen)
prefixes_v4(id, prefix, mask_length, first_seen, last_seen)
prefixes_v6(id, prefix, mask_length, first_seen, last_seen)
-- Live routing tables (one per IP version)
live_routes_v4(id, prefix, mask_length, origin_asn, peer_ip, as_path,
next_hop, last_updated)
live_routes_v6(id, prefix, mask_length, origin_asn, peer_ip, as_path,
next_hop, last_updated)
-- Relationship tracking
peerings(id, as_a, as_b, first_seen, last_seen)
bgp_peers(id, peer_ip, peer_asn, last_message_type, last_seen)
```
## Configuration
Configuration is handled via environment variables and OS-specific paths:
| Variable | Default | Description |
|----------|----------|-------------|
| `PORT` | `8080` | HTTP server port, a whole number from 1 to 65535 |
| `DEBUG` | (empty) | Set to `routewatch` for debug logging |
| `XDG_DATA_HOME` | `/var/lib` (in the Docker image) | Base of the state directory; must be an absolute path |
| `GOMEMLIMIT` | `1536MiB` (in the Docker image) | Go soft memory limit; see Memory |
| `MALLOC_ARENA_MAX` | `2` (in the Docker image) | glibc malloc arena cap, a positive whole number; see Memory |
A variable that is set to an invalid value stops the start with an error and a
non-zero exit. An empty variable counts as unset.
State directory (database location):
- macOS: `~/Library/Application Support/routewatch/`
- Linux: `/var/lib/berlin.sneak.app.routewatch/` when running as root,
otherwise `$XDG_DATA_HOME/berlin.sneak.app.routewatch/` (with `XDG_DATA_HOME`
unset, `~/.local/share/berlin.sneak.app.routewatch/`). In the Docker image
this is `/var/lib/berlin.sneak.app.routewatch/`.
## Memory
The daemon holds a live routing table, so its memory grows with the size of the
data it tracks. The image sets ceilings that keep it inside a 5 GiB container.
Budget:
- Go heap: a 1.5 GiB soft limit (`GOMEMLIMIT=1536MiB`, set in the image).
- SQLite: at most 640 MiB of page cache across the connection pool (64 MiB per
connection, 10 connections) and a 1.5 GiB hard heap limit for the C library.
- glibc allocator: the SQLite C library runs on glibc `malloc`, which frees
page-cache chunks back to per-arena free lists rather than to the kernel, so
process RSS tracks the high-water mark of those arenas, not SQLite's live
heap. glibc creates up to eight arenas per core, so on a many-core host the
retained memory — and thus RSS — grows with the core count. The image sets
`MALLOC_ARENA_MAX=2` to bound it; the two-arena cap costs nothing here because
database writes are already serialized.
- About 0.2 GiB for everything else in the runtime.
Run the container with a memory limit of 5 GiB and swap disabled:
```bash
docker run --memory=5g --memory-swap=5g ...
```
or the equivalent in your deployment tool. This leaves headroom above the
ceilings for spikes and the kernel page cache.
Override the Go soft limit by setting `GOMEMLIMIT` in the environment (for
example `-e GOMEMLIMIT=1GiB`); this replaces the image default. `MALLOC_ARENA_MAX`
can be overridden the same way, but raising it lets RSS climb again on a
many-core host.
What happens at each limit:
- Go soft limit: as the heap approaches `GOMEMLIMIT`, the runtime runs garbage
collection more aggressively rather than growing further.
- SQLite: at a 1 GiB soft heap limit it recycles its page cache instead of
allocating more; at the 1.5 GiB hard heap limit a statement fails with an
out-of-memory error, and the handler logs it and drops that batch. The process
keeps running.
- Handler queues: each of the four handler queues holds at most 20,000 messages.
When a queue fills, the streamer drops messages instead of blocking.
With `DEBUG=routewatch` the daemon logs a `System stats` line every 60 seconds
with the goroutine count and Go memory figures.
## Running under upaas
What the [upaas](https://git.eeqj.de/sneak/upaas) app needs:
- Container port: `8080`.
- Volume: one, at container path `/var/lib/berlin.sneak.app.routewatch`.
- Environment: nothing is required. Leave `XDG_DATA_HOME`, `GOMEMLIMIT` and
`MALLOC_ARENA_MAX` at the image's values. `DEBUG=routewatch` is optional and
adds the `System stats` memory line to the log.
- Memory Limit: `5g`, the 5 GiB limit from Memory above. upaas sets no swap
limit, so on a host with swap Docker allows the same amount of swap again.
- Health check: the image's `HEALTHCHECK` requests
`/.well-known/healthcheck.json` on the container port. upaas reads the
container's health 60 seconds after a deploy and fails the deploy unless it
is `healthy`.
## Development
```bash
# Run tests
make test
# Format code
make fmt
# Run linter
make lint
# Build
make
```
## Entrypoints
This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard: normalized scripts in `script/` are the entrypoints for the
development workflow, and the Makefile targets are thin shims that call
them. We provide:
- `script/bootstrap` — install all development dependencies (git, make, go, Go
module download); warns when docker is missing
- `script/setup` — make a fresh clone ready for development: runs
`script/bootstrap`, then `script/install-precommit`
- `script/projectname` — print the project name (used for the Docker
image tag)
- `script/test` — build the `test` phase alone, which runs
`go test -short -timeout 90s -race -cover ./...` (verbose rerun on failure).
The `-short` flag skips the live-network integration test so the run is
deterministic and offline.
- `script/lint` — build the `lint` phase alone, which runs golangci-lint
v2.14.0 with `.golangci.yml`
- `script/fmt` — format all code (writes)
- `script/fmt-check` — check formatting (read-only)
- `script/check` — run `script/test`, `script/lint`, and
`script/fmt-check`
- `script/docker` — build the Docker image tagged via
`script/projectname`
- `script/cibuild` — CI entrypoint: runs `script/bootstrap` and
`script/check`, then builds the image
- `script/precommit` — pre-commit gate: `go mod tidy` + `go fmt` (must
not change files), then `script/check`
- `script/install-precommit` — install the git pre-commit hook that
runs `script/precommit`
Tests and the linter run only in Docker, as the `test` and `lint` phases of the
`Dockerfile`; the image cannot be built unless both pass. Every `docker build`
in `script/` passes `--no-cache`, so each run executes every check.
The live-network integration test `TestRouteWatchLiveFeed` streams the RIPE
RIS feed for a few seconds and is skipped in short mode. To run it on demand,
invoke `go test` directly without `-short`:
`go test -run TestRouteWatchLiveFeed ./internal/routewatch/`.
## License
MIT. See [`LICENSE`](LICENSE).
## Author
[@sneak](https://sneak.berlin)
REPO_POLICIES.md
+679
View File
@@ -0,0 +1,679 @@
---
title: Repository Policies
last_modified: 2026-10-04
---
This document covers repository structure, tooling, and workflow standards. Code
style conventions are in separate documents:
- [Code Styleguide](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE.md)
(general, bash, Docker)
- [Go](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE_GO.md)
- [JavaScript](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE_JS.md)
- [Python](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE_PYTHON.md)
- [Go HTTP Server Conventions](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/GO_HTTP_SERVER_CONVENTIONS.md)
---
- Cross-project documentation (such as this file) must include
`last_modified: YYYY-MM-DD` in the YAML front matter so it can be kept in sync
with the authoritative source as policies evolve.
- **ALL external references must be pinned by cryptographic hash.** This
includes Docker base images, Go modules, npm packages, GitHub Actions, and
anything else fetched from a remote source. Version tags (`@v4`, `@latest`,
`:3.21`, etc.) are server-mutable and therefore remote code execution
vulnerabilities. The ONLY acceptable way to reference an external dependency
is by its content hash (Docker `@sha256:...`, Go module hash in `go.sum`, npm
integrity hash in lockfile, GitHub Actions `@<commit-sha>`). No exceptions.
This also means never `curl | bash` to install tools like pyenv, nvm, rustup,
etc. Instead, download a specific release archive from GitHub, verify its hash
(hardcoded in the Dockerfile or script), and only then install. Unverified
install scripts are arbitrary remote code execution. This is the single most
important rule in this document. Double-check every external reference in
every file before committing. There are zero exceptions to this rule.
- Every repo with software must have a root `Makefile` with these targets:
`make bootstrap`, `make setup`, `make test`, `make lint`, `make fmt` (writes),
`make fmt-check` (read-only), `make check` (runs `test`, `lint`, `fmt-check`),
`make docker`, and `make hooks` (installs pre-commit hook). A model Makefile
is at `https://git.eeqj.de/sneak/prompts/raw/branch/main/Makefile`.
- Repos follow the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
pattern: the implementation of each Makefile target lives in an executable
script in `script/` (`script/bootstrap`, `script/setup`, `script/test`,
`script/lint`, `script/fmt`, `script/fmt-check`, `script/check`,
`script/docker`), and the Makefile targets are thin shims that call them. The
scripts must be POSIX sh (`#!/bin/sh`, `set -eu`, no bashisms) so they run in
minimal containers (e.g. alpine images have no bash); locate the repo root
with `$(cd "$(dirname "$0")/.." && pwd -P)` and `cd` there before acting. From
the standard's canonical set we use `bootstrap`, `setup` (make the repo ready
for development after a fresh clone: runs `bootstrap`, then
`install-precommit`, plus any repo-specific initialization), `test`, and
`cibuild`. `script/bootstrap` installs all dependencies idempotently and
assumes nothing is present: base tools come from nix, apt, brew, or apk
(detected in that order; apt runs noninteractive). For node it uses the
installed node if present; otherwise it installs a PINNED node version via
nvm, first installing nvm itself if missing — from a hash-verified GitHub
release archive (never `curl | sh`), with bash installed as an explicit
prerequisite since nvm requires bash. yarn is then pinned via
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
always exact versions. `script/cibuild` runs the CI build: it changes to the
repo root, runs `script/bootstrap`, runs `script/check`, and builds the image
with the version; the Gitea workflow calls it. **`script/cibuild` runs
`script/bootstrap` first**, because the workflow checks out the repo and runs
nothing else, while `script/fmt-check` runs the formatter on the host: on a
pristine checkout with nothing installed the run dies there, after the
containerised gates have passed. **The bootstrap alone is not enough**:
`script/bootstrap` installs node and yarn under nvm and leaves neither on the
`PATH` of the shell that called it, so a bare `yarn` still exits 127. The host
entrypoints that need yarn — `script/fmt` and `script/fmt-check` — therefore
source nvm for the pinned node version before invoking it, exactly as
`script/bootstrap`'s own install step does. A runner carrying nothing but
docker and git then gets through `script/check`. Four further scripts are our
own extensions to the standard: `script/check` runs `script/test`,
`script/lint` and `script/fmt-check`; `script/precommit` is what the git
pre-commit hook runs, and it calls `script/check`; `script/install-precommit`
installs the git pre-commit hook (the `make hooks` target shims to it); and
`script/projectname` (literally that filename) simply outputs the project's
name. Scripts that need the name call `script/projectname` — e.g.
`script/docker` assembles its image tag from it — so those scripts stay
byte-identical across all repos. Repo-type-specific pre-commit extras (e.g.
`go mod tidy` verification in Go repos) belong in `script/precommit`, not in
the hook itself. Model scripts are at
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
must document the provided scripts in an **Entrypoints** section (see the
README requirements below).
- Always use Makefile targets (`make fmt`, `make test`, `make lint`, etc.)
instead of invoking the underlying tools directly. The Makefile is the single
source of truth for how these operations are run.
- The Makefile is authoritative documentation for how the repo is used. Beyond
the required targets above, it should have targets for every common operation:
running a local development server (`make run`, `make dev`), re-initializing
or migrating the database (`make db-reset`, `make migrate`), building
artifacts (`make build`), generating code, seeding data, or anything else a
developer would do regularly. If someone checks out the repo and types
`make<tab>`, they should see every meaningful operation available. A new
contributor should be able to understand the entire development workflow by
reading the Makefile.
- Every repo should have a `Dockerfile`, and it carries the repo's gates: a
`lint` phase and a `test` phase, with the final stage depending on both so the
image cannot be built unless they pass. For non-server repos the final stage
brings up a development environment; for server repos it is the runtime image.
The gate phases and the build stage start from their pinned base images and
install what those images lack either inline, as the canonical Go `Dockerfile`
below does for `git`, or by running `script/bootstrap`, as the `prompts`
repo's own `Dockerfile` does for its yarn packages. The development
environment stage installs development prerequisites by running
`script/bootstrap` rather than duplicating its installs inline. A stage that
runs `script/bootstrap` COPYs `script/` and the dependency manifests
(`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before running it.
- **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is
no separate lint file. `script/lint` and `script/test` each build one phase
and nothing else:
```sh
docker build --no-cache --target lint -t "$(script/projectname)-lint" .
docker build --no-cache --target test -t "$(script/projectname)-test" .
```
**A stage that is not the last one in the file is built only when the final
stage's chain depends on it, or when `--target` names it.** That is why the
two gates are always invoked by name here, and why the final stage carries a
`COPY --from=` of a harmless file from each of them: without that edge a
plain `docker build .` builds the last stage alone and exits 0 having linted
and tested nothing.
**Every `docker build` in `script/` is tagged**, here and in
`script/cibuild` and `script/docker`. An untagged build leaves a dangling
image behind on every invocation, on every developer host and every CI
runner; a tagged one replaces the previous image.
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
`eslint`, `prettier` — never through `make lint` or `script/test`, which are
themselves a `docker build` and would recurse into a daemon that does not
exist in a build step. Formatting is the exception and stays on the host:
`script/fmt` writes the working tree, and `script/fmt-check` is its
read-only twin.
**No lint verdict may come from a host invocation of the linter.** On a
shared host golangci-lint reads a result cache keyed on file content rather
than location, so a second checkout of the same content is served the first
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
exit non-zero with `parallel golangci-lint is running` — a status a caller
cannot tell from real findings. Both have produced wrong verdicts in this
org, in both directions. A container has its own cache, its own `TMPDIR` and
a digest-pinned binary, so neither is reachable.
- **Any build that runs checks is built with `--no-cache`.** Docker invalidates
a `COPY` layer only when the copied content changes, so on an unchanged tree
the check `RUN` is served from cache, nothing executes, and the build still
exits 0. Every `docker build` in `script/` therefore passes `--no-cache`:
`script/lint`, `script/test`, `script/cibuild` and `script/docker` are the
four, and there is no fifth — `script/check` runs the two gate phases and
`script/fmt-check`, and builds no image of its own. A bare `docker build .` is
not evidence that anything ran: a sub-second build reporting success is a
cache hit, not a result. Never invalidate by pruning — `docker builder prune`
and friends destroy a build cache shared with every other build on the host.
When a check is added or changed, prove it works by planting a defect it must
catch and watching the run fail on it, then revert the defect. A green run
alone shows neither that the check ran nor that it covers what it should.
- **The gate phases are separate stages, and the build stage depends on both.**
The lint phase is based on the `golangci/golangci-lint` image (pinned by
hash), so lint failures surface in seconds rather than after a full compile,
and the test phase is based on the Debian Go image. The canonical Go repo
`Dockerfile`:
```dockerfile
# Lint phase
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
FROM golangci/golangci-lint@sha256:... AS lint
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN golangci-lint run --config .golangci.yml ./...
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
# image ships and the alpine one does not.
# golang:1.x, YYYY-MM-DD
FROM golang@sha256:... AS test
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go test -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from either phase above; the copies
# are what make BuildKit build them first, so this stage cannot run
# unless lint and test passed.
# golang:1.x-alpine, YYYY-MM-DD
FROM golang@sha256:... AS builder
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
# The VERSION build arg when one is given, otherwise
# `git describe --tags --always` on the .git in the build context. With
# .git present, a version that is still empty, dev or unknown fails the
# build: git is missing or could not read the checkout.
ARG VERSION
RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ]; then \
case "$VERSION" in ""|dev|unknown) \
echo "version is '$VERSION' although .git is present" >&2; \
exit 1 ;; \
esac; \
fi; \
CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \
-o /app ./cmd/app/
# Runtime stage, and the last one
FROM alpine@sha256:...
COPY --from=builder /app /usr/local/bin/app
ENTRYPOINT ["app"]
```
Key points:
- The lint phase uses the `golangci/golangci-lint` image directly (it has
both Go and the linter), so nothing needs installing.
- `COPY --from=<phase> /src/go.sum /dev/null` is a no-op copy whose only
purpose is the ordering edge. BuildKit runs stages in parallel by default,
and a stage nothing depends on is not built at all, so without these two
lines a red gate would not fail the build.
- Keep the runtime stage last, and if you add a stage after it, give it the
same two copies. A plain `docker build .` builds the last stage's chain
and nothing else.
- If the project uses `//go:embed` directives that reference build artifacts
(e.g. a web frontend compiled in a separate stage), the lint phase must
create placeholder files so the embed directives resolve. Example:
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
- If the project requires CGO or system libraries for linting, install them
in the lint phase. The `golangci/golangci-lint` image is Debian-based and
has no `apk`, so install with `apt-get` under the Debian package name
(`libvips-dev`, where alpine says `vips-dev`), and delete the package
lists in the same `RUN`, so the layer does not keep them:
```dockerfile
RUN apt-get update \
&& apt-get install -y --no-install-recommends libvips-dev \
&& rm -rf /var/lib/apt/lists/*
```
- `.dockerignore` lets `.git` into the build context. It keeps out every git
`config` at any depth (`**/.git/config`, `**/.git/modules/**/config`): the
repository's own, each submodule's under `.git/modules/`, and that of a
submodule keeping its own `.git` directory. `git describe` does not need
them, and each can hold a credential: a password in a remote URL, or the
token the CI checkout step stores there. A submodule whose name has a
`config` segment (`config`, `deploy/config`, `config/lib`) loses its whole
git directory to `**/.git/modules/**/config`, and Go's version stamping
then fails the build: give it a name without that segment
(`git submodule add --name`). The stage that compiles has `git` (the
Debian Go image has it; an alpine one needs `apk add --no-cache git`) and
takes the version from the `VERSION` build argument when one is given,
otherwise from `git describe --tags --always`. That gives the tag on a
tagged commit; on a later commit, the tag, the number of commits since it
and the short commit (`v1.2.3-4-gabc1234`); and the short commit when no
tag is reachable. The stage that compiles also marks its working directory
safe for git (`git config --system --add safe.directory /src`): a context
sent as a tar stream keeps the sender's file owners, and git refuses a
checkout owned by another user, so the version would come out empty.
`ARG VERSION` has no default, and the build fails if the context carries
`.git` and the version still comes out empty, `dev` or `unknown`. A plain
`docker build .` with no build arguments must succeed; a Dockerfile that
refuses an empty build argument drops that refusal and keeps the argument.
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
runs `script/cibuild` on push, and checks out the repo as its only other step.
That script bootstraps, runs the gate phases, and then builds the image, so a
successful run means every check passed; a bare `docker build .` does not
carry the same guarantee, because its gate phases may come from the cache. The
image build is uncached and so runs the gate phases a second time. That is the
price of the rule above, and it is worth paying: the image that ships is built
from a run of its own gates rather than from a cache entry. A separate
workflow limited to `main` by a `branches` list under `on: push` cannot be
checked by review: to try a change to it, add the feature branch to that list
and push, then remove the branch from the list again before merging. Keep any
job in it that publishes behind `if: github.ref_name == 'main'`, so the run
from the feature branch publishes nothing.
- Use platform-standard formatters: `black` for Python, `prettier` for
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
two exceptions: four-space indents (except Go), and `proseWrap: always` for
Markdown (hard-wrap at 80 columns). Documentation and writing repos (Markdown,
HTML, CSS) should also have `.prettierrc` and `.prettierignore`.
- Pre-commit hook: runs `script/precommit`, which calls `script/check`. If local
testing is not possible in the repo, `script/precommit` may skip `script/test`
and run only `script/lint` and `script/fmt-check`. The hook is installed by
`script/install-precommit`; the Makefile must provide a `make hooks` target
that shims to it.
- All repos with software must have tests that run via the platform-standard
test framework (`go test`, `pytest`, `jest`/`vitest`, etc.). If no meaningful
tests exist yet, add the most minimal test possible — e.g. importing the
module under test to verify it compiles/parses. There is no excuse for
`make test` to be a no-op.
- `make test` must complete in under 60 seconds. That is the hard cap, and a
suite that exceeds it fails. Under 20 seconds is the target. A suite between
20 and 60 seconds is still green, but the overage must be filed as an
improvement bug against that repo. Add a 90-second timeout to the test
invocation (`go test -timeout 90s`). The backstop deliberately sits above the
hard cap so that it catches a genuinely hung test rather than a merely slow
one.
- **The test command should use the conditional verbose rerun pattern.** Run
tests without `-v` (verbose) first. If tests fail, automatically rerun with
`-v` to show full output. This keeps CI logs and `docker build` output clean
on success (just package/suite summaries) while providing full diagnostic
detail on failure (every test case, every assertion). The command lives in the
`test` phase of the `Dockerfile`, since `script/test` builds that phase; the
Makefile form below is the same pattern for any repo-local invocation:
```makefile
test:
@<test-command> || \
{ echo "--- Rerunning with -v for details ---"; \
<test-command-with-v>; exit 1; }
```
Go example:
```makefile
test:
@go test -count=1 -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -count=1 -timeout 90s -race -v ./...; exit 1; }
```
`-count=1` is required on both invocations: it defeats Go's test _result_
cache, so neither run can report a stored pass in place of running the
tests. It leaves the build cache alone, so it costs the runtime of the suite
and no recompilation.
That cache is Go's own, separate from Docker's layer cache. Go stores a
passing result in its cache directory (`GOCACHE`), and when the same tests
run again on unchanged code it prints that result, marked `(cached)`,
without running them. That matters on a developer's machine, where this
target runs and the directory lasts from one run to the next. The `test`
phase of the `Dockerfile` needs no `-count=1`: its base image holds no
result for this repo's tests and nothing before its `go test` step runs a
test, so there is nothing to replay. `--no-cache` (above) is what makes that
step run on an unchanged tree.
Python example:
```makefile
test:
@python -m pytest || \
{ echo "--- Rerunning with -v for details ---"; \
python -m pytest -v; exit 1; }
```
The `exit 1` ensures the target always fails after a rerun — the first run
already proved the tests are broken, so the build must not pass even if a
flaky test happens to succeed on the second attempt. The rerun exists solely
for diagnostic output.
- Docker builds must complete in under 5 minutes.
- `make check` must not modify any files in the repo. Tests may use temporary
directories.
- `main` must always pass `make check`, no exceptions.
- Never commit secrets. `.env` files, credentials, API keys, and private keys
must be in `.gitignore`. No exceptions.
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
language build artifacts, and `node_modules/`. Fetch the standard `.gitignore`
from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when
setting up a new repo. These patterns are written to `.gitignore`'s own
semantics, in which an unanchored pattern already matches at every depth; they
are not a `.dockerignore` and must not be transplanted into one unmodified.
- **`.dockerignore` does not use `.gitignore` semantics, and copying patterns
across unmodified leaves secrets in the build context.** Docker matches with
`moby/patternmatcher`: `filepath.Match` semantics plus a `**` extension, so
`*` does not cross `/` and a pattern without a leading `**/` is anchored at
the build-context root. A `.dockerignore` listing `.env`, `*.pem` and `*.key`
therefore excludes only the copies at the repository root, while `config/.env`
and `certs/server.key` still reach the context and can land in an image layer
— which is more dangerous than a short file with no secret patterns at all,
because it reads as solved and stops anyone looking. Give every
depth-independent pattern the `**/` prefix and leave only genuinely
root-anchored entries unprefixed: `.claude`, and the repo's own host-built
binary, written `/myapp` and never `**/myapp`, which would also match
`cmd/myapp/` and delete the package directory from the context. Matching is
case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so
secret names use character ranges — `**/*.[kK][eE][yY]`, `**/*.[pP][eE][mM]`,
and likewise for `.envrc` and the extensionless SSH keys. Where such a pattern
also catches something the build needs, re-include it with a negation
(`!docs/example.env`); deleting the pattern reopens the exposure for every
other file it covers. Fetch the standard `.dockerignore` from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.dockerignore` and extend
it with the repo's own artifacts.
- **In-repo agent scratch belongs in both files, written to each file's own
semantics.** `.claude/` holds one worktree per in-flight agent — an entire
additional checkout of the repo — so under `COPY . .` the build context
inflates by a multiple of the repo and another session's unreviewed work can
be copied into an image layer. In `.gitignore` the entry is `.claude/`,
unanchored. In `.dockerignore` it is `.claude`, anchored and with **no** `**/`
prefix, because the prefixed form would also delete any nested directory of
that name from the build. Anchoring carries a known gap that the canonical
`.dockerignore` states in its own comment, since consuming repos receive the
file and not the tracker: the directory is created in the agent's working
directory, so a repo running agents in subdirectories still ships
`services/api/.claude/` and must add its own anchored entry there.
- **A plain `docker build .` of a clone stamps the version that
`git describe --tags --always` gives**, derived from the `.git` in the build
context as the canonical `Dockerfile` above shows. Without its failure check,
a missing `git` or an unreadable checkout would leave `-X main.Version=` empty
and the build would still exit 0. `script/docker` and `script/cibuild` pass
the version they compute on the host; it takes precedence. They do this
byte-identically across repos:
```sh
# Own line: a failing command substitution inside an argument does not
# trip `set -e`, so the inline form degrades to an empty constant.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$(script/projectname)" .
```
`--always` makes an untagged repo yield an abbreviated commit hash rather
than failing, and the `[ -n "$version" ]` line is the single place the
fallback is applied — a live check that fires on a build from an export with
no `.git` and on a repository with no commits yet. Do not fold it into the
substitution as `|| echo unknown`, which makes the guard unreachable. The
Dockerfile's side is `ARG VERSION` in the stage that compiles, declared
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
the scripts stay byte-identical. One consequence for CI: the standard
checkout action clones shallow and fetches no tags, so a repo that embeds a
tag-derived version must set `fetch-depth: 0` on its checkout step.
- **Verify `.dockerignore` by enumerating the image, not by reading the
patterns.** Plant files at the root _and_ at least two directories deep, build
a probe image that does `COPY . .`, and list what actually landed
(`docker run --rm --entrypoint find IMAGE /app`). The `transferring context`
size is not a substitute: a nested secret is a few bytes, and BuildKit
transfers only the delta from the previous build.
- **No build artifacts in version control.** Code-derived data (compiled
bundles, minified output, generated assets) must never be committed to the
repository if it can be avoided. The build process (e.g. Dockerfile, Makefile)
should generate these at build time. Notable exception: Go protobuf generated
files (`.pb.go`) ARE committed because repos need to work with `go get`, which
downloads code but does not execute code generation.
- Never use `git add -A` or `git add .`. Always stage files explicitly by name.
- Never force-push to `main`.
- Make all changes on a feature branch. You can do whatever you want on a
feature branch.
- `.golangci.yml` is standardized. The vendored copy in a consuming repo must
_NEVER_ be modified by an agent: fetch it from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml` and keep it
byte-identical, so that no repo can quietly loosen its own linting. Linter
configuration changes are made to the canonical copy in the `prompts` repo and
reach consuming repos by re-vendoring; an agent may open a PR against
canonical, which only the user merges. One list is exempt from byte-identity,
because it cannot be written once for every repo: the `deny` list of the
`test-support` depguard rule, where a repo names its own test-support packages
by full import path. A repo adds entries there and changes nothing else, and a
re-vendor carries its entries forward. The canonical golangci-lint version is
v2.14.0 (released 2026-09-24), pinned as the digest of the lint phase's base
image
(`golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f`,
which reports `2.14.0 built with go1.27.0 from 114493f9`). A module's `go`
directive must not name a newer Go minor version than the one golangci-lint
was built with, or golangci-lint refuses to lint it: this release lints
`go 1.27.1` but not `go 1.28`. That digest is the only pin, since no repo
installs golangci-lint on the host. A repo sets the lint phase digest to the
one named here and re-vendors `.golangci.yml` in the same commit, whichever of
the two prompted the change: the canonical copy can name linters that an older
golangci-lint rejects, and a newer golangci-lint can add linters that
`default: all` switches on until the canonical copy disables them.
- **`script/bootstrap` installs a pinned tool by comparing versions, never by
testing presence.** An `if ! command -v <tool>; then install; fi` guard tests
`PATH` only, so on an already-provisioned machine the pin is inert and a
version bump is a silent no-op — while the Dockerfile, installing into a clean
image, gets the pinned version, so a local `make check` and `make docker` can
disagree about what the tool even is. The canonical form:
- compares the installed version against the pin over the **whole** version
token; a parser that stops at the first `-` reports `2.12.2` for a host
running `2.12.2-rc1` and skips the install;
- treats absent, non-zero, empty or unrecognised `--version` output as a
mismatch, so the failure direction is a redundant install and never a
skipped one;
- after installing, re-resolves the binary the way callers do — `hash -r`,
then through `PATH`, not through the directory the installer wrote to —
and fails naming the resolved path, since an install that a shadowing
binary hides succeeds while changing nothing any caller sees;
- is actually called, and prints the version on both success paths: a
function defined and never invoked has the same exit status and the same
empty output as one that worked.
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
A Go tool a repo needs on the host is installed with `go install` pinned to
a commit hash (`go install <package>@<commit hash>`). It is never tracked as
a `go.mod` tool dependency or through a `tools.go` file, either of which
pulls the tool's own dependencies into the repo's `go.mod` and `go.sum`.
- When pinning images or packages by hash, add a comment above the reference
with the version and date (YYYY-MM-DD).
- Use `yarn`, not `npm`.
- Write all dates as YYYY-MM-DD (ISO 8601).
- Simple projects should be configured with environment variables.
- Dockerized web services listen on port 8080 by default, overridable with
`PORT`.
- **HTTP/web services must be hardened for production internet exposure before
tagging 1.0.** This means full compliance with security best practices
including, without limitation, all of the following:
- **Security headers** on every response:
- `Strict-Transport-Security` (HSTS) with `max-age` of at least one year
and `includeSubDomains`.
- `Content-Security-Policy` (CSP) with a restrictive default policy
(`default-src 'self'` as a baseline, tightened per-resource as
needed). Never use `unsafe-inline` or `unsafe-eval` unless
unavoidable, and document the reason.
- `X-Frame-Options: DENY` (or `SAMEORIGIN` if framing is required).
Prefer the `frame-ancestors` CSP directive as the primary control.
- `X-Content-Type-Options: nosniff`.
- `Referrer-Policy: strict-origin-when-cross-origin` (or stricter).
- `Permissions-Policy` restricting access to browser features the
application does not use (camera, microphone, geolocation, etc.).
- **Request and response limits:**
- Maximum request body size enforced on all endpoints (e.g. Go
`http.MaxBytesReader`). Choose a sane default per-route; never accept
unbounded input.
- Maximum response body size where applicable (e.g. paginated APIs).
- `ReadTimeout` and `ReadHeaderTimeout` on the `http.Server` to defend
against slowloris attacks.
- `WriteTimeout` on the `http.Server`.
- `IdleTimeout` on the `http.Server`.
- Per-handler execution time limits via `context.WithTimeout` or
chi/stdlib `middleware.Timeout`.
- **Authentication and session security:**
- Rate limiting on password-based authentication endpoints. API keys are
high-entropy and not susceptible to brute force, so they are exempt.
- CSRF tokens on all state-mutating HTML forms. API endpoints
authenticated via `Authorization` header (Bearer token, API key) are
exempt because the browser does not attach these automatically.
- Passwords stored using bcrypt, scrypt, or argon2 — never plain-text,
MD5, or SHA.
- Session cookies set with `HttpOnly`, `Secure`, and `SameSite=Lax` (or
`Strict`) attributes.
- **Reverse proxy awareness:**
- True client IP detection when behind a reverse proxy
(`X-Forwarded-For`, `X-Real-IP`). The application must accept
forwarded headers only from a configured set of trusted proxy
addresses — never trust `X-Forwarded-For` unconditionally.
- **CORS:**
- Authenticated endpoints must restrict `Access-Control-Allow-Origin` to
an explicit allowlist of known origins. Wildcard (`*`) is acceptable
only for public, unauthenticated read-only APIs.
- **Error handling:**
- Internal errors must never leak stack traces, SQL queries, file paths,
or other implementation details to the client. Return generic error
messages in production; detailed errors only when `DEBUG` is enabled.
- **TLS:**
- Services never terminate TLS directly. They are always deployed behind
a TLS-terminating reverse proxy. The service itself listens on plain
HTTP. However, HSTS headers and `Secure` cookie flags must still be
set by the application so that the browser enforces HTTPS end-to-end.
This list is non-exhaustive. Apply defense-in-depth: if a standard security
hardening measure exists for HTTP services and is not listed here, it is
still expected. When in doubt, harden.
- `README.md` is the primary documentation. Required sections:
- **Description**: First line must include the project name, purpose,
category (web server, SPA, CLI tool, etc.), license, and author. Example:
"µPaaS is an MIT-licensed Go web application by @sneak that receives
git-frontend webhooks and deploys applications via Docker in realtime."
- **Getting Started**: Copy-pasteable install/usage code block.
- **Entrypoints**: Opens by stating that the repo adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard (with that link), then documents each provided `script/`
entrypoint and its purpose.
- **Rationale**: Why does this exist?
- **Design**: How is the program structured?
- **TODO**: Update meticulously, even between commits. When planning, put
the todo list in the README so a new agent can pick up where the last one
left off.
- **License**: MIT, GPL, or WTFPL. Ask the user for new projects. Include a
`LICENSE` file in the repo root and a License section in the README.
- **Author**: [@sneak](https://sneak.berlin).
- First commit of a new repo should contain only `README.md`.
- Go module root: `sneak.berlin/go/<name>`. Always run `go mod tidy` before
committing.
- Use SemVer.
- Database migrations live in `internal/db/migrations/` and must be embedded in
the binary.
- `000_migration.sql` — contains ONLY the creation of the migrations
tracking table itself. Nothing else.
- `001_schema.sql` — the full application schema.
- **Pre-1.0.0:** never add additional migration files (002, 003, etc.).
There is no installed base to migrate. Edit `001_schema.sql` directly.
- **Post-1.0.0:** add new numbered migration files for each schema change.
Never edit existing migrations after release.
- All repos should have an `.editorconfig` enforcing the project's indentation
settings.
- Avoid putting files in the repo root unless necessary. Root should contain
only project-level config files (`README.md`, `AGENTS.md`, `Makefile`,
`Dockerfile`, `LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`,
and language-specific config). Everything else goes in a subdirectory.
Canonical subdirectory names:
- `bin/` — executable scripts and tools
- `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose
body is a single call into `internal/` or `pkg/`, no project logic in
`cmd/`
- `configs/` — configuration templates and examples
- `deploy/` — deployment manifests (k8s, compose, terraform)
- `docs/` — documentation and markdown (README.md stays in root)
- `internal/` — Go internal packages
- `internal/db/migrations/` — database migrations
- `pkg/` — Go library packages
- `share/` — systemd units, data files
- `static/` — static assets (images, fonts, etc.)
- `web/` — web frontend source
- When setting up a new repo, files from the `prompts` repo may be used as
templates. Fetch them from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/<path>`.
- New repos must contain at minimum:
- `README.md`, `.git`, `.gitignore`, `.editorconfig`
- `LICENSE`, `REPO_POLICIES.md` (copy from the `prompts` repo)
- `Makefile`
- `script/` entrypoints (`bootstrap`, `setup`, `projectname`, `test`,
`lint`, `fmt`, `fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
`install-precommit`)
- `Dockerfile`, `.dockerignore`
- `.gitea/workflows/check.yml`
- Go: `go.mod`, `go.sum`, `.golangci.yml`
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
- Python: `pyproject.toml`
- Guidance for coding agents lives in one `AGENTS.md` at the repository root. It
is never committed under a file or directory named after one agent tool, such
as `CLAUDE.md` or `.claude/`, and never split into separate memory files.
TODO.md
+127
View File
@@ -0,0 +1,127 @@
# Workflow
* branch (from `main`)
* do the work in Next Step
* move Next Step to the top of Completed Steps
* move the top item of Future Steps into Next Step
* commit (`TODO.md` changes in the same commit as the work)
* merge to `main` if the branch is not protected, otherwise open a PR
* push
# Status
pre-1.0. No git tags. The linter and the tests are phases of the Docker
build, and the Gitea workflow runs `script/cibuild` (the format check, both
phases, then the image) on every push. The
image sets memory ceilings for a 5 GiB container (README "Memory") and the
README says how to run it under upaas (README "Running under upaas"). A
35-hour run of `3898daa` on the live feed peaked at about 1 GiB, without a
container memory limit.
# Next Step
`next` waits for sneak to merge it to `main` through
https://git.eeqj.de/sneak/routewatch/pulls/6. After that, setting
routewatch up under upaas on fsn1app1 and deploying it are his
(https://git.eeqj.de/sneak/routewatch/issues/31), and so is the run under a
real 5 GiB limit (https://git.eeqj.de/sneak/routewatch/issues/3).
# Completed Steps
- 2026-10-14: the shared files are re-vendored from `sneak/prompts` at
`dd4027b`: golangci-lint v2.14.0 and its configuration, lint and test as
separate uncached Dockerfile phases, agent guidance in `AGENTS.md`; the code
is fixed for what the new linter configuration finds (closes #52)
- 2026-10-03: `/api/v1/stats` serves the prefix distribution from memory,
seeded at startup and adjusted on every live-route write, so a request no
longer reads every live route (closes #30)
- 2026-10-03: a new database is created with `auto_vacuum` set to
incremental, through the connection string so it is set before the file
is first written, and each periodic incremental vacuum now returns up to
1000 free pages; the late `PRAGMA auto_vacuum` in `Initialize`, which
SQLite ignored, is gone (closes #43)
- 2026-10-03: looking up an IP address no longer reads every IPv6 route:
both families find the most specific live route with at most 33 or 129
lookups on the prefix index, and the IPv4 range columns are gone. Prefixes
from the feed are stored in one text form, so an IPv6 withdrawal, which the
feed sends uncompressed, now removes its route (closes #48)
- 2026-10-02: a plain `docker build .` stamps the commit's tag or short
commit (`git describe --tags --always`) into the page footer instead of
`unknown`: `.dockerignore` sends `.git` without `.git/config`, a `VERSION`
build argument takes precedence, and `make build` stamps the same value
(closes #46)
- 2026-09-29: the entrypoint creates the data directory if it is missing and
stops the start if a step fails; README "Running under upaas" no longer
asks for the host directory to be created first (closes #42)
- 2026-09-29: `.dockerignore` keeps `.git`, local build output, local
databases and `.env` out of the Docker build context, and so out of the
source archive in the image (closes #39)
- 2026-09-29: README first line names the MIT license and the author;
the License section now says MIT and links `LICENSE`, and an Author
section was added; this file brought up to date (closes #38)
- 2026-09-29: MIT `LICENSE` (closes #1)
- 2026-09-28: stopping the daemon while the feed is flowing no longer
panics with "send on closed channel": the read loop checks for a stop
just before handing a message to the handler queues, and a second
`Stop` no longer closes the queues again (closes #34)
- 2026-09-28: `docker stop` no longer kills the daemon 2 seconds after the
stop signal: the entrypoint switches to the `routewatch` user with
`setpriv` instead of `runuser`, so the daemon receives the signal itself
and gets the whole wait `docker stop` allows, up to its own 60-second
limit (closes #33)
- 2026-09-28: ready to run under upaas: a set but invalid `PORT`,
`XDG_DATA_HOME` or `MALLOC_ARENA_MAX` stops the start, the health
check follows `PORT`, README "Running under upaas" section (closes
#31)
- 2026-09-22: realtime in-memory database statistics: counts seeded at
startup and adjusted on every write, oldest/newest route timestamps via
index-end lookups; `/api/v1/stats` no longer scans the tables (closes
#27)
- 2026-09-21: batch writes take the write lock when their transaction
begins (`_txlock=immediate`), so they wait out a WAL checkpoint instead
of failing with "database is locked" (closes #25)
- 2026-09-21: `MALLOC_ARENA_MAX=2` in the image caps glibc malloc arenas,
so memory outside the Go runtime no longer grows with the core count
(closes #23)
- 2026-09-21: `GOMEMLIMIT=1536MiB` in the image; README Memory section
with the memory budget and the 5 GiB container limit (closes #13)
- 2026-09-21: the four handler queues hold at most 20,000 messages each,
down from 100,000 (closes #11)
- 2026-09-21: two goroutine leaks fixed: the stats handlers after a
timeout and the streamer's tickers on every reconnect (closes #12)
- 2026-09-21: parsed RIS messages no longer keep the unused `Community`
and `Raw` fields (closes #9)
- 2026-09-21: `.editorconfig`, and a Gitea workflow that runs
`script/cibuild` on every push (closes #14)
- 2026-09-21: the peering handler's AS-path map holds at most 500,000
paths and is swapped for an empty one every 30 seconds instead of copied
(closes #10)
- 2026-09-21: SQLite memory bounded across the whole connection pool: a
64 MiB page cache on each connection, 1 GiB soft and 1.5 GiB hard heap
limits (closes #8)
- 2026-09-21: the Docker build runs the format check, the linter and the
tests, linting in a separate stage on a golangci-lint image pinned by
digest (closes #5)
- 2026-09-21: `make test` skips the live-network feed test (`-short`), so
`make check` no longer depends on the network (closes #2)
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints,
Makefile shims, README Entrypoints section
- 2026-02-22: repo policy compliance: required policy files, .gitignore
update, Makefile fmt-check/check/docker/hooks targets, gofmt pass
(repo-policies-compliance, unmerged)
- 2026-01-01: WAL checkpointing: periodic (5s), on startup with logging,
TRUNCATE mode; fixed slow queries
- 2025-12-31: status page: navbar, home page with search, oldest/newest
route timestamps, NULL handling in WHOIS stats; container runs as
routewatch user with proper state dir
- 2025-12-30: structured HTTP request logging, increased timeouts,
CIDR prefix URL routing fixes, status page metrics and footer
- 2025-12-29: Dockerfile multi-stage build with source archive; SQLite
incremental vacuum for non-blocking space reclamation
- 2025-07-27: initial RouteWatch BGP stream monitor
# Future Steps
- Production memory under 5 GiB: whether to test under a real 5 GiB
container limit on fsn1app1 is open for sneak
(https://git.eeqj.de/sneak/routewatch/issues/3)
cmd/asinfo-gen/main.go
+22 -7
View File
@@ -2,8 +2,10 @@
package main
import (
"context"
"encoding/csv"
"encoding/json"
"errors"
"io"
"log"
"net/http"
@@ -27,13 +29,17 @@ func main() {
// Fetch the CSV data
log.Println("Fetching AS CSV data...")
resp, err := http.Get(asCsvURL)
req, err := http.NewRequestWithContext(
context.Background(), http.MethodGet, asCsvURL, nil)
if err != nil {
log.Fatalf("Failed to fetch CSV: %v", err)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
log.Fatalf("Failed to fetch CSV: %v", err)
}
defer func() {
_ = resp.Body.Close()
}()
if resp.StatusCode != http.StatusOK {
log.Fatalf("Failed to fetch CSV: HTTP %d", resp.StatusCode)
@@ -48,17 +54,20 @@ func main() {
log.Fatalf("Failed to read CSV header: %v", err)
}
if len(header) != 3 || header[0] != "asn" || header[1] != "handle" || header[2] != "description" {
if len(header) != 3 || header[0] != "asn" || header[1] != "handle" ||
header[2] != "description" {
log.Fatalf("Unexpected CSV header: %v", header)
}
// Read all records
var asInfos []ASInfo
for {
record, err := reader.Read()
if err == io.EOF {
if errors.Is(err, io.EOF) {
break
}
if err != nil {
log.Printf("Error reading CSV record: %v", err)
@@ -86,12 +95,18 @@ func main() {
})
}
// Closed here rather than deferred: log.Fatalf exits without running
// deferred calls.
_ = resp.Body.Close()
log.Printf("Parsed %d AS records", len(asInfos))
// Convert to JSON and output to stdout
encoder := json.NewEncoder(os.Stdout)
encoder.SetIndent("", "\t")
if err := encoder.Encode(asInfos); err != nil {
err = encoder.Encode(asInfos)
if err != nil {
log.Fatalf("Failed to encode JSON: %v", err)
}
}
cmd/streamdumper/main.go
+13 -12
View File
@@ -9,16 +9,14 @@ import (
"os/signal"
"syscall"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/streamer"
"log/slog"
)
func main() {
// Set up logger to only show errors
logger := slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{
Level: slog.LevelError,
}))
// Set up logger
logger := logger.New()
// Create metrics tracker
metricsTracker := metrics.New()
@@ -28,9 +26,17 @@ func main() {
// Register raw message handler that prints to stdout
s.RegisterRawHandler(func(line string) {
fmt.Println(line)
_, _ = fmt.Fprintln(os.Stdout, line)
})
// Start streaming before any defer: log.Fatal exits without running
// deferred calls.
err := s.Start(context.Background())
if err != nil {
log.Fatal("Failed to start streamer:", err)
}
defer s.Stop()
// Set up context with cancellation
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
@@ -38,18 +44,13 @@ func main() {
// Handle signals
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, os.Interrupt, syscall.SIGTERM)
go func() {
<-sigChan
log.Println("Received shutdown signal")
cancel()
}()
// Start streaming
if err := s.Start(); err != nil {
log.Fatal("Failed to start streamer:", err)
}
defer s.Stop()
// Wait for context cancellation
<-ctx.Done()
}
entrypoint.sh
+20
View File
@@ -0,0 +1,20 @@
#!/bin/bash
set -euo pipefail
# glibc silently ignores a malformed MALLOC_ARENA_MAX, so refuse it here.
if [[ -n "${MALLOC_ARENA_MAX:-}" && ! "$MALLOC_ARENA_MAX" =~ ^[1-9][0-9]*$ ]]; then
echo "MALLOC_ARENA_MAX must be a positive whole number, got '$MALLOC_ARENA_MAX'" >&2
exit 1
fi
# Give the data directory to the routewatch user before the daemon starts,
# whether it is missing, an empty root-owned mount, or holds another uid's files.
mkdir -p /var/lib/berlin.sneak.app.routewatch
cd /var/lib/berlin.sneak.app.routewatch
chown -R routewatch:routewatch .
chmod 700 .
# setpriv replaces itself with the daemon, so the daemon receives the stop
# signal directly. runuser would stay in between and kill the daemon 2 seconds
# after passing the signal on.
exec setpriv --reuid=routewatch --regid=routewatch --init-groups -- /app/routewatch
go.mod
+2
View File
@@ -3,11 +3,13 @@ module git.eeqj.de/sneak/routewatch
go 1.24.4
require (
github.com/dustin/go-humanize v1.0.1
github.com/go-chi/chi/v5 v5.2.2
github.com/google/uuid v1.6.0
github.com/mattn/go-sqlite3 v1.14.29
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9
go.uber.org/fx v1.24.0
golang.org/x/term v0.33.0
)
require (
go.sum
+4
View File
@@ -1,5 +1,7 @@
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/go-chi/chi/v5 v5.2.2 h1:CMwsvRVTbXVytCk1Wd72Zy1LAsAh9GxMmSNWLHCG618=
github.com/go-chi/chi/v5 v5.2.2/go.mod h1:L2yAIGWB3H+phAw1NxKwWM+7eUH/lU8pOMm5hHcoops=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
@@ -24,5 +26,7 @@ go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo=
go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so=
golang.org/x/sys v0.34.0 h1:H5Y5sJ2L2JRdyv7ROF1he/lPdvFsd0mJHFw2ThKHxLA=
golang.org/x/sys v0.34.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/term v0.33.0 h1:NuFncQrRcaRvVmgRkvM3j/F00gWIAlcmlB8ACEKmGIg=
golang.org/x/term v0.33.0/go.mod h1:s18+ql9tYWp1IfpV9DmCtQDDSRBUjKaw9M1eAv5UeF0=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
internal/config/config.go
+70 -4
View File
@@ -2,10 +2,12 @@
package config
import (
"errors"
"fmt"
"os"
"path/filepath"
"runtime"
"strconv"
"time"
)
@@ -15,6 +17,21 @@ const (
// dirPermissions for creating directories
dirPermissions = 0750 // rwxr-x---
// defaultRouteExpirationMinutes is the default route expiration timeout in minutes
defaultRouteExpirationMinutes = 5
// defaultPort is the HTTP port used when PORT is not set
defaultPort = 8080
// maxPort is the highest TCP port number
maxPort = 65535
)
var (
errRelativeXDGDataHome = errors.New("XDG_DATA_HOME must be an absolute path")
errUnsupportedOS = errors.New("unsupported operating system")
errInvalidPort = errors.New("PORT must be a whole number")
)
// Config holds configuration for the entire application
@@ -22,8 +39,21 @@ type Config struct {
// StateDir is the directory for all application state (database, snapshots)
StateDir string
// Port is the TCP port the HTTP server listens on
Port int
// MaxRuntime is the maximum runtime (0 = run forever)
MaxRuntime time.Duration
// EnableBatchedDatabaseWrites enables batched database operations for
// better performance
EnableBatchedDatabaseWrites bool
// RouteExpirationTimeout is how long a route can go without being
// refreshed before expiring
// Default is 2 hours which is conservative for BGP (typical BGP hold
// time is 90-180 seconds)
RouteExpirationTimeout time.Duration
}
// New creates a new Config with default paths based on the OS
@@ -33,9 +63,20 @@ func New() (*Config, error) {
return nil, fmt.Errorf("failed to determine state directory: %w", err)
}
port, err := getPort()
if err != nil {
return nil, err
}
return &Config{
StateDir: stateDir,
MaxRuntime: 0, // Run forever by default
Port: port,
// Run forever by default
MaxRuntime: 0,
// Enable batching by default
EnableBatchedDatabaseWrites: true,
// For active route monitoring
RouteExpirationTimeout: defaultRouteExpirationMinutes * time.Minute,
}, nil
}
@@ -57,13 +98,20 @@ func getStateDirectory() (string, error) {
return filepath.Join(home, "Library", "Application Support", AppIdentifier), nil
case "linux", "freebsd", "openbsd", "netbsd":
// The XDG spec requires an absolute path; a relative one would put
// the database somewhere unexpected.
xdgData := os.Getenv("XDG_DATA_HOME")
if xdgData != "" && !filepath.IsAbs(xdgData) {
return "", fmt.Errorf("%w, got %q", errRelativeXDGDataHome, xdgData)
}
// Unix-like: /var/lib/berlin.sneak.app.routewatch if root, else XDG_DATA_HOME
if os.Geteuid() == 0 {
return filepath.Join("/var/lib", AppIdentifier), nil
}
// Check XDG_DATA_HOME first
if xdgData := os.Getenv("XDG_DATA_HOME"); xdgData != "" {
if xdgData != "" {
return filepath.Join(xdgData, AppIdentifier), nil
}
@@ -76,14 +124,32 @@ func getStateDirectory() (string, error) {
return filepath.Join(home, ".local", "share", AppIdentifier), nil
default:
return "", fmt.Errorf("unsupported operating system: %s", runtime.GOOS)
return "", fmt.Errorf("%w: %s", errUnsupportedOS, runtime.GOOS)
}
}
// getPort returns the HTTP port from PORT, or defaultPort when PORT is not set
func getPort() (int, error) {
value := os.Getenv("PORT")
if value == "" {
return defaultPort, nil
}
// ParseUint, unlike Atoi, refuses a sign: the health check URL cannot use "+9090"
port, err := strconv.ParseUint(value, 10, 0)
if err != nil || port < 1 || port > maxPort {
return 0, fmt.Errorf("%w from 1 to %d, got %q",
errInvalidPort, maxPort, value)
}
return int(port), nil
}
// EnsureDirectories creates all necessary directories if they don't exist
func (c *Config) EnsureDirectories() error {
// Ensure state directory exists
if err := os.MkdirAll(c.StateDir, dirPermissions); err != nil {
err := os.MkdirAll(c.StateDir, dirPermissions)
if err != nil {
return fmt.Errorf("failed to create state directory: %w", err)
}
internal/config/config_internal_test.go
+67
View File
@@ -0,0 +1,67 @@
package config
import (
"runtime"
"testing"
)
func TestNewReadsPort(t *testing.T) {
tests := map[string]int{
"": defaultPort,
"1": 1,
"9090": 9090,
"65535": 65535,
}
for value, want := range tests {
t.Run(value, func(t *testing.T) {
t.Setenv("PORT", value)
t.Setenv("XDG_DATA_HOME", "")
cfg, err := New()
if err != nil {
t.Fatalf("New() with PORT=%q: %v", value, err)
}
if cfg.Port != want {
t.Errorf("New() with PORT=%q: Port = %d, want %d", value, cfg.Port, want)
}
})
}
}
func TestNewRefusesInvalidPort(t *testing.T) {
for _, value := range []string{"0", "65536", "-1", "+9090", "http", "80.5"} {
t.Run(value, func(t *testing.T) {
t.Setenv("PORT", value)
t.Setenv("XDG_DATA_HOME", "")
_, err := New()
if err == nil {
t.Errorf("New() with PORT=%q returned no error", value)
}
})
}
}
func TestNewRefusesRelativeXDGDataHome(t *testing.T) {
if runtime.GOOS == "darwin" {
t.Skip("macOS does not read XDG_DATA_HOME")
}
t.Setenv("PORT", "")
t.Setenv("XDG_DATA_HOME", "relative/path")
_, err := New()
if err == nil {
t.Error("New() with a relative XDG_DATA_HOME returned no error")
}
t.Setenv("XDG_DATA_HOME", "/var/lib")
_, err = New()
if err != nil {
t.Errorf("New() with XDG_DATA_HOME=/var/lib: %v", err)
}
}
internal/database/counts.go
+218
View File
@@ -0,0 +1,218 @@
package database
import (
"context"
"fmt"
"sync"
)
// liveCounts holds the running row counts and the prefix distribution that the
// stats endpoints report. They are seeded once at startup from the tables and
// then adjusted on every write, so a stats read serves them from memory instead
// of running a query over the tables. The COUNT(*) scans (issue 27) and then
// the prefix distribution query (issue 30) each grew with the database until
// they took the whole request timeout and made /api/v1/stats return 500.
//
// A single mutex guards all fields so the stats reader takes a consistent
// snapshot at one instant and writers, which already run under the database
// write lock, adjust the counts after their transaction commits.
type liveCounts struct {
mu sync.RWMutex
asns int
prefixesV4 int
prefixesV6 int
peerings int
peers int
routesV4 int
routesV6 int
// The prefix distribution: for each mask length, the number of distinct
// prefixes that have at least one live route.
distributionV4 [ipv4Bits + 1]int
distributionV6 [ipv6Bits + 1]int
}
// seed sets every count to the value read from the tables at startup. It runs
// before any writer, so it needs no coordination with the adjust methods.
func (c *liveCounts) seed(
asns, prefixesV4, prefixesV6, peerings, peers, routesV4, routesV6 int,
distributionV4, distributionV6 []PrefixDistribution,
) {
c.mu.Lock()
defer c.mu.Unlock()
c.asns = asns
c.prefixesV4 = prefixesV4
c.prefixesV6 = prefixesV6
c.peerings = peerings
c.peers = peers
c.routesV4 = routesV4
c.routesV6 = routesV6
for _, entry := range distributionV4 {
addAtMaskLength(c.distributionV4[:], entry.MaskLength, entry.Count)
}
for _, entry := range distributionV6 {
addAtMaskLength(c.distributionV6[:], entry.MaskLength, entry.Count)
}
}
// addASNs adds n to the ASN count.
func (c *liveCounts) addASNs(n int) {
c.mu.Lock()
c.asns += n
c.mu.Unlock()
}
// addPrefixes adds to the IPv4 and IPv6 prefix counts.
func (c *liveCounts) addPrefixes(v4, v6 int) {
c.mu.Lock()
c.prefixesV4 += v4
c.prefixesV6 += v6
c.mu.Unlock()
}
// addPeerings adds n to the peering count.
func (c *liveCounts) addPeerings(n int) {
c.mu.Lock()
c.peerings += n
c.mu.Unlock()
}
// addPeers adds n to the BGP peer count.
func (c *liveCounts) addPeers(n int) {
c.mu.Lock()
c.peers += n
c.mu.Unlock()
}
// addRoutes adds to the IPv4 and IPv6 live-route counts. Deletions pass
// negative values.
func (c *liveCounts) addRoutes(v4, v6 int) {
c.mu.Lock()
c.routesV4 += v4
c.routesV6 += v6
c.mu.Unlock()
}
// addToDistribution adds n to the IPv4 and IPv6 prefix distributions once for
// each listed mask length. A write lists the mask lengths of the prefixes it
// gave their first live route with n = 1, and of the prefixes it left with no
// live route with n = -1.
func (c *liveCounts) addToDistribution(maskLengthsV4, maskLengthsV6 []int, n int) {
c.mu.Lock()
defer c.mu.Unlock()
for _, maskLength := range maskLengthsV4 {
addAtMaskLength(c.distributionV4[:], maskLength, n)
}
for _, maskLength := range maskLengthsV6 {
addAtMaskLength(c.distributionV6[:], maskLength, n)
}
}
// addAtMaskLength adds n to counts[maskLength]. A mask length the array has no
// entry for is ignored, so a malformed route cannot crash the daemon.
func addAtMaskLength(counts []int, maskLength, n int) {
if maskLength >= 0 && maskLength < len(counts) {
counts[maskLength] += n
}
}
// fill copies the counts into a Stats, including the derived totals, under a
// single read lock so the reader sees one consistent snapshot.
func (c *liveCounts) fill(s *Stats) {
c.mu.RLock()
defer c.mu.RUnlock()
s.ASNs = c.asns
s.IPv4Prefixes = c.prefixesV4
s.IPv6Prefixes = c.prefixesV6
s.Prefixes = c.prefixesV4 + c.prefixesV6
s.Peerings = c.peerings
s.Peers = c.peers
s.IPv4Routes = c.routesV4
s.IPv6Routes = c.routesV6
s.LiveRoutes = c.routesV4 + c.routesV6
s.IPv4PrefixDistribution = distributionList(c.distributionV4[:])
s.IPv6PrefixDistribution = distributionList(c.distributionV6[:])
}
// distributionList lists the mask lengths that have at least one prefix, in
// ascending order, the way the distribution query returns them.
func distributionList(counts []int) []PrefixDistribution {
var list []PrefixDistribution
for maskLength, count := range counts {
if count > 0 {
list = append(list,
PrefixDistribution{MaskLength: maskLength, Count: count})
}
}
return list
}
// countRows returns the number of rows in the named table. It is used only at
// startup to seed the in-memory counters, so a full COUNT(*) is acceptable.
func (d *Database) countRows(ctx context.Context, table string) (int, error) {
var n int
// table is one of a fixed set of literals below, never external input.
err := d.db.QueryRowContext(ctx, "SELECT COUNT(*) FROM "+table).Scan(&n)
if err != nil {
return 0, fmt.Errorf("failed to count %s: %w", table, err)
}
return n, nil
}
// seedCounts reads the current row counts and prefix distribution from the
// tables into the in-memory counters. It runs once at startup, before the
// streamer begins writing.
func (d *Database) seedCounts(ctx context.Context) error {
asns, err := d.countRows(ctx, "asns")
if err != nil {
return err
}
prefixesV4, err := d.countRows(ctx, "prefixes_v4")
if err != nil {
return err
}
prefixesV6, err := d.countRows(ctx, "prefixes_v6")
if err != nil {
return err
}
peerings, err := d.countRows(ctx, "peerings")
if err != nil {
return err
}
peers, err := d.countRows(ctx, "bgp_peers")
if err != nil {
return err
}
routesV4, err := d.countRows(ctx, tableLiveRoutesV4)
if err != nil {
return err
}
routesV6, err := d.countRows(ctx, tableLiveRoutesV6)
if err != nil {
return err
}
distributionV4, distributionV6, err := d.GetPrefixDistributionContext(ctx)
if err != nil {
return err
}
d.counts.seed(asns, prefixesV4, prefixesV6, peerings, peers, routesV4, routesV6,
distributionV4, distributionV6)
return nil
}
internal/database/counts_internal_test.go
+614
View File
@@ -0,0 +1,614 @@
package database
import (
"context"
"slices"
"sync"
"testing"
"time"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/logger"
"github.com/google/uuid"
)
// testV4PeerIP is the peer the IPv4 routes in these tests are learned from.
const testV4PeerIP = "192.0.2.1"
// testV6Prefix is the prefix of every IPv6 route in these tests.
const testV6Prefix = "2001:db8::/32"
// mkV4Route builds an IPv4 live route with its mask length taken from the
// prefix.
func mkV4Route(t *testing.T, prefix string, asn int, ts time.Time) *LiveRoute {
t.Helper()
maskLength, err := prefixMaskLength(prefix)
if err != nil {
t.Fatalf("prefixMaskLength(%s): %v", prefix, err)
}
return &LiveRoute{
ID: uuid.New(),
Prefix: prefix,
MaskLength: maskLength,
IPVersion: ipVersionV4,
OriginASN: asn,
PeerIP: testV4PeerIP,
ASPath: []int{asn},
NextHop: "192.0.2.254",
LastUpdated: ts,
}
}
// mkV6Route builds an IPv6 live route for testV6Prefix.
func mkV6Route(asn int, ts time.Time) *LiveRoute {
return &LiveRoute{
ID: uuid.New(),
Prefix: testV6Prefix,
MaskLength: 32,
IPVersion: ipVersionV6,
OriginASN: asn,
PeerIP: "2001:db8::1",
ASPath: []int{asn},
NextHop: "2001:db8::ffff",
LastUpdated: ts,
}
}
// TestLiveCountsTrackWritesInRealtime checks that the stats counts start at
// zero, reflect each write the moment it commits (no recompute, no timer), do
// not move when a route is merely re-announced, and drop when a route is
// deleted. These counts are what /api/v1/stats reports; before this change the
// endpoint recomputed them with a COUNT(*) over each table on every request.
func TestLiveCountsTrackWritesInRealtime(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ctx := context.Background()
empty, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext on empty database: %v", err)
}
if empty.ASNs != 0 || empty.Prefixes != 0 || empty.Peerings != 0 ||
empty.Peers != 0 || empty.LiveRoutes != 0 {
t.Fatalf("empty database counts nonzero: %+v", empty)
}
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
writeASNsPrefixesAndPeers(t, db, ts)
routes := []*LiveRoute{
mkV4Route(t, "198.51.100.0/24", 64500, ts),
mkV4Route(t, "203.0.113.0/24", 64501, ts.Add(time.Minute)),
mkV6Route(64502, ts.Add(2*time.Minute)),
}
err = db.UpsertLiveRouteBatch(routes)
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
stats, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext: %v", err)
}
assertCounts(t, "after inserts", stats, wantCounts{
asns: 2, prefixes: 2, peerings: 1, peers: 1,
ipv4Routes: 2, ipv6Routes: 1, liveRoutes: 3,
})
// Re-announcing the same routes is an update, not an insert: counts hold.
err = db.UpsertLiveRouteBatch(routes)
if err != nil {
t.Fatalf("UpsertLiveRouteBatch (re-announce): %v", err)
}
stats, err = db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext: %v", err)
}
assertCounts(t, "after re-announce", stats, wantCounts{
asns: 2, prefixes: 2, peerings: 1, peers: 1,
ipv4Routes: 2, ipv6Routes: 1, liveRoutes: 3,
})
// A withdrawal removes one route.
err = db.DeleteLiveRouteBatch([]LiveRouteDeletion{{
Prefix: "203.0.113.0/24", OriginASN: 64501,
PeerIP: testV4PeerIP, IPVersion: ipVersionV4,
}})
if err != nil {
t.Fatalf("DeleteLiveRouteBatch: %v", err)
}
stats, err = db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext: %v", err)
}
assertCounts(t, "after delete", stats, wantCounts{
asns: 2, prefixes: 2, peerings: 1, peers: 1,
ipv4Routes: 1, ipv6Routes: 1, liveRoutes: 2,
})
}
// writeASNsPrefixesAndPeers writes two ASNs, two prefixes, one peer and one
// peering: the rows besides live routes that the stats count.
func writeASNsPrefixesAndPeers(t *testing.T, db *Database, ts time.Time) {
t.Helper()
err := db.GetOrCreateASNBatch(map[int]time.Time{64500: ts, 64501: ts})
if err != nil {
t.Fatalf("GetOrCreateASNBatch: %v", err)
}
err = db.UpdatePrefixesBatch(map[string]time.Time{
"198.51.100.0/24": ts,
testV6Prefix: ts,
})
if err != nil {
t.Fatalf("UpdatePrefixesBatch: %v", err)
}
err = db.UpdatePeerBatch(map[string]PeerUpdate{
testV4PeerIP: {
PeerIP: testV4PeerIP, PeerASN: 64500,
MessageType: "UPDATE", Timestamp: ts,
},
})
if err != nil {
t.Fatalf("UpdatePeerBatch: %v", err)
}
err = db.RecordPeering(64500, 64501, ts)
if err != nil {
t.Fatalf("RecordPeering: %v", err)
}
}
// TestLiveCountsSeededFromDatabaseAtStartup writes rows, reopens the same
// database file, and checks the counts and the prefix distribution come back
// from the seed scan rather than starting at zero.
func TestLiveCountsSeededFromDatabaseAtStartup(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
err = db.GetOrCreateASNBatch(map[int]time.Time{64500: ts, 64501: ts, 64502: ts})
if err != nil {
t.Fatalf("GetOrCreateASNBatch: %v", err)
}
err = db.UpsertLiveRouteBatch([]*LiveRoute{
mkV4Route(t, "198.51.100.0/24", 64500, ts),
mkV6Route(64502, ts),
})
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
err = db.Close()
if err != nil {
t.Fatalf("Close: %v", err)
}
reopened, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to reopen database: %v", err)
}
defer func() { _ = reopened.Close() }()
stats, err := reopened.GetStatsContext(context.Background())
if err != nil {
t.Fatalf("GetStatsContext after reopen: %v", err)
}
if stats.ASNs != 3 {
t.Errorf("seeded ASNs = %d, want 3", stats.ASNs)
}
if stats.IPv4Routes != 1 || stats.IPv6Routes != 1 || stats.LiveRoutes != 2 {
t.Errorf("seeded routes = (v4 %d, v6 %d, total %d), want (1, 1, 2)",
stats.IPv4Routes, stats.IPv6Routes, stats.LiveRoutes)
}
assertDistribution(t, "seeded IPv4 distribution", stats.IPv4PrefixDistribution,
[]PrefixDistribution{{MaskLength: 24, Count: 1}})
assertDistribution(t, "seeded IPv6 distribution", stats.IPv6PrefixDistribution,
[]PrefixDistribution{{MaskLength: 32, Count: 1}})
}
// TestPrefixDistributionTracksWrites checks that the prefix distribution the
// stats read reports stays exact across each kind of live-route write, and that
// after every step it equals what the distribution query reads from the route
// tables. The steps run once through the batch methods the prefix handler uses
// and once through the single-route methods.
func TestPrefixDistributionTracksWrites(t *testing.T) {
t.Parallel()
steps := prefixDistributionSteps(t)
for _, batch := range []bool{true, false} {
name := "single-route writes"
if batch {
name = "batch writes"
}
t.Run(name, func(t *testing.T) {
t.Parallel()
db, err := New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ctx := context.Background()
for _, step := range steps {
err := announceRoutes(db, batch, step.announce)
if err != nil {
t.Fatalf("%s: announce: %v", step.name, err)
}
err = withdrawRoutes(db, batch, step.withdraw)
if err != nil {
t.Fatalf("%s: withdraw: %v", step.name, err)
}
stats, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("%s: GetStatsContext: %v", step.name, err)
}
queryV4, queryV6, err := db.GetPrefixDistributionContext(ctx)
if err != nil {
t.Fatalf("%s: GetPrefixDistributionContext: %v", step.name, err)
}
assertDistribution(t, step.name+": IPv4 distribution",
stats.IPv4PrefixDistribution, step.wantV4)
assertDistribution(t, step.name+": IPv6 distribution",
stats.IPv6PrefixDistribution, step.wantV6)
assertDistribution(t, step.name+": IPv4 distribution query",
queryV4, step.wantV4)
assertDistribution(t, step.name+": IPv6 distribution query",
queryV6, step.wantV6)
}
})
}
}
// prefixDistributionStep is one step of TestPrefixDistributionTracksWrites:
// the routes it announces, then the routes it withdraws, then the
// distributions both the stats and the distribution query must report.
type prefixDistributionStep struct {
name string
announce []*LiveRoute
withdraw []*LiveRoute
wantV4 []PrefixDistribution
wantV6 []PrefixDistribution
}
// prefixDistributionSteps returns the steps of
// TestPrefixDistributionTracksWrites, in the order they run.
func prefixDistributionSteps(t *testing.T) []prefixDistributionStep {
t.Helper()
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
shared := mkV4Route(t, "198.51.100.0/24", 64500, ts)
sharedSecondPeer := mkV4Route(t, "198.51.100.0/24", 64500, ts)
sharedSecondPeer.PeerIP = "192.0.2.2"
other := mkV4Route(t, "203.0.113.0/24", 64501, ts)
wide := mkV4Route(t, "172.16.0.0/16", 64502, ts)
v6 := mkV6Route(64503, ts)
v6SecondPeer := mkV6Route(64503, ts)
v6SecondPeer.PeerIP = "2001:db8::2"
all := []PrefixDistribution{{MaskLength: 16, Count: 1}, {MaskLength: 24, Count: 2}}
v6Only := []PrefixDistribution{{MaskLength: 32, Count: 1}}
return []prefixDistributionStep{
{
name: "new routes",
announce: []*LiveRoute{shared, other, wide, v6},
wantV4: all, wantV6: v6Only,
},
{
name: "re-announcement",
announce: []*LiveRoute{shared, other, wide, v6},
wantV4: all, wantV6: v6Only,
},
{
name: "second peer announces a prefix that has a live route",
announce: []*LiveRoute{sharedSecondPeer, v6SecondPeer},
wantV4: all, wantV6: v6Only,
},
{
name: "withdrawal of a route that is not the last for its prefix",
withdraw: []*LiveRoute{shared, v6},
wantV4: all, wantV6: v6Only,
},
{
name: "withdrawal of the last route for a prefix",
withdraw: []*LiveRoute{sharedSecondPeer, v6SecondPeer},
wantV4: []PrefixDistribution{
{MaskLength: 16, Count: 1}, {MaskLength: 24, Count: 1},
},
},
{
name: "withdrawal of every remaining route",
withdraw: []*LiveRoute{other, wide},
},
{
name: "two peers announce a new prefix together",
announce: []*LiveRoute{shared, sharedSecondPeer, v6, v6SecondPeer},
wantV4: []PrefixDistribution{{MaskLength: 24, Count: 1}},
wantV6: v6Only,
},
{
name: "both routes for a prefix withdrawn together",
withdraw: []*LiveRoute{shared, sharedSecondPeer, v6, v6SecondPeer},
},
// The feed often withdraws a route that is not live. That must not take
// the prefix out of the distribution. A count wrongly taken below zero is
// left out of the answer, so the next step shows it: its announcement
// at the same mask length would then not be counted.
{
name: "withdrawal of routes that are not live",
withdraw: []*LiveRoute{other, v6},
},
{
name: "announcement after a withdrawal of routes that are not live",
announce: []*LiveRoute{other, v6},
wantV4: []PrefixDistribution{{MaskLength: 24, Count: 1}},
wantV6: v6Only,
},
}
}
// announceRoutes writes routes in one UpsertLiveRouteBatch, or with one
// UpsertLiveRoute each.
func announceRoutes(db *Database, batch bool, routes []*LiveRoute) error {
if batch {
return db.UpsertLiveRouteBatch(routes)
}
for _, route := range routes {
err := db.UpsertLiveRoute(route)
if err != nil {
return err
}
}
return nil
}
// withdrawRoutes removes routes in one DeleteLiveRouteBatch, or with one
// DeleteLiveRoute each. It names each route by prefix and peer only, with no
// origin ASN, as a withdrawal from the feed does when its message carries no AS
// path.
func withdrawRoutes(db *Database, batch bool, routes []*LiveRoute) error {
if !batch {
for _, route := range routes {
err := db.DeleteLiveRoute(route.Prefix, 0, route.PeerIP)
if err != nil {
return err
}
}
return nil
}
deletions := make([]LiveRouteDeletion, 0, len(routes))
for _, route := range routes {
deletions = append(deletions, LiveRouteDeletion{
Prefix: route.Prefix, PeerIP: route.PeerIP, IPVersion: route.IPVersion,
})
}
return db.DeleteLiveRouteBatch(deletions)
}
// TestStatsRouteTimestamps checks the oldest/newest route timestamps are read
// from the right rows across both tables and parse into time.Time. The old
// MIN/MAX union query read its result into *time.Time, which the driver could
// not parse, so it logged a warning every call and left both timestamps nil.
func TestStatsRouteTimestamps(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ctx := context.Background()
empty, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext on empty database: %v", err)
}
if empty.OldestRoute != nil || empty.NewestRoute != nil {
t.Fatalf("empty database timestamps = (%v, %v), want (nil, nil)",
empty.OldestRoute, empty.NewestRoute)
}
base := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
oldest := base
newest := base.Add(2 * time.Minute)
err = db.UpsertLiveRouteBatch([]*LiveRoute{
mkV4Route(t, "198.51.100.0/24", 64500, base.Add(time.Minute)),
mkV4Route(t, "203.0.113.0/24", 64501, oldest),
mkV6Route(64502, newest),
})
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
stats, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext: %v", err)
}
if stats.OldestRoute == nil || !stats.OldestRoute.Equal(oldest) {
t.Errorf("OldestRoute = %v, want %v", stats.OldestRoute, oldest)
}
if stats.NewestRoute == nil || !stats.NewestRoute.Equal(newest) {
t.Errorf("NewestRoute = %v, want %v", stats.NewestRoute, newest)
}
}
// TestLiveCountsConcurrentReadWrite runs writers and stats readers at once so
// the race detector proves the counters are safe under concurrent use.
func TestLiveCountsConcurrentReadWrite(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
const writers = 4
var wg sync.WaitGroup
wg.Add(writers)
for w := range writers {
go func(base int) {
defer wg.Done()
for i := range 25 {
asn := 65000 + base*100 + i
route := mkV6Route(asn, ts)
route.PeerIP = "2001:db8::" + uuid.NewString()
err := db.UpsertLiveRoute(route)
if err != nil {
t.Errorf("UpsertLiveRoute: %v", err)
return
}
}
}(w)
}
var readerWG sync.WaitGroup
readerWG.Add(1)
stop := make(chan struct{})
go func() {
defer readerWG.Done()
for {
select {
case <-stop:
return
default:
_, err := db.GetStatsContext(context.Background())
if err != nil {
t.Errorf("GetStatsContext: %v", err)
return
}
}
}
}()
wg.Wait()
close(stop)
readerWG.Wait()
stats, err := db.GetStatsContext(context.Background())
if err != nil {
t.Fatalf("GetStatsContext: %v", err)
}
if want := writers * 25; stats.IPv6Routes != want {
t.Errorf("IPv6Routes = %d, want %d", stats.IPv6Routes, want)
}
// Every writer announced the same prefix, so it counts once.
assertDistribution(t, "IPv6 distribution", stats.IPv6PrefixDistribution,
[]PrefixDistribution{{MaskLength: 32, Count: 1}})
}
type wantCounts struct {
asns int
prefixes int
peerings int
peers int
ipv4Routes int
ipv6Routes int
liveRoutes int
}
func assertCounts(t *testing.T, when string, got Stats, want wantCounts) {
t.Helper()
if got.ASNs != want.asns {
t.Errorf("%s: ASNs = %d, want %d", when, got.ASNs, want.asns)
}
if got.Prefixes != want.prefixes {
t.Errorf("%s: Prefixes = %d, want %d", when, got.Prefixes, want.prefixes)
}
if got.Peerings != want.peerings {
t.Errorf("%s: Peerings = %d, want %d", when, got.Peerings, want.peerings)
}
if got.Peers != want.peers {
t.Errorf("%s: Peers = %d, want %d", when, got.Peers, want.peers)
}
if got.IPv4Routes != want.ipv4Routes {
t.Errorf("%s: IPv4Routes = %d, want %d", when, got.IPv4Routes, want.ipv4Routes)
}
if got.IPv6Routes != want.ipv6Routes {
t.Errorf("%s: IPv6Routes = %d, want %d", when, got.IPv6Routes, want.ipv6Routes)
}
if got.LiveRoutes != want.liveRoutes {
t.Errorf("%s: LiveRoutes = %d, want %d", when, got.LiveRoutes, want.liveRoutes)
}
}
func assertDistribution(t *testing.T, what string, got, want []PrefixDistribution) {
t.Helper()
if !slices.Equal(got, want) {
t.Errorf("%s = %v, want %v", what, got, want)
}
}
internal/database/database.go
+2125 -161
View File
File diff suppressed because it is too large Load Diff
internal/database/database_internal_test.go
+434
View File
@@ -0,0 +1,434 @@
package database
import (
"context"
"database/sql"
"errors"
"fmt"
"net/netip"
"sync"
"testing"
"time"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/logger"
"github.com/google/uuid"
)
// tempStoreMemory is the PRAGMA temp_store value meaning "hold temp B-trees in
// memory"; the DSN change must leave temp_store below this so they spill to disk.
const tempStoreMemory = 2
// autoVacuumIncremental is the PRAGMA auto_vacuum value meaning "incremental".
const autoVacuumIncremental = 2
// vacuumTestRoutes is how many routes the vacuum test writes and then deletes,
// enough to leave many free pages in the file.
const vacuumTestRoutes = 2000
// heldConnections is how many pooled connections the pragma test holds open at
// once so each is a distinct SQLite connection that parsed the DSN.
const heldConnections = 5
// testPeerIP is the peer every route in the IP lookup test is learned from.
const testPeerIP = "192.0.2.254"
// Parameters for the checkpoint-contention regression test.
const (
// contentionIterations is how many batch writes race the checkpoint loop.
contentionIterations = 400
// contendedASNCount is the small set of ASNs the batches reuse, so most
// batches update existing rows and exercise the read-before-write path.
contendedASNCount = 16
// asnSecondBand offsets a second ASN per batch so each batch writes more
// than one row.
asnSecondBand = 100
)
// TestGetIPInfoFindsMostSpecificLiveRoute stores nested live prefixes for both
// families and checks that a lookup returns the most specific one covering the
// address, ErrNoRoute when none covers it, and the next less specific prefix
// once the only route of the most specific one is withdrawn.
func TestGetIPInfoFindsMostSpecificLiveRoute(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
// Nested live prefixes, each originated by its own AS.
origins := map[string]int{
"10.0.0.0/8": 64500,
"10.1.0.0/16": 64501,
"10.1.2.0/24": 64502,
"2001:db8::/32": 64500,
"2001:db8:1::/48": 64501,
"2001:db8:1:2::/64": 64502,
}
writeOriginRoutes(t, db, origins)
// lookup checks that ip resolves to the live prefix want, or to ErrNoRoute
// when want is empty.
lookup := func(ip, want string) {
t.Helper()
info, err := db.GetIPInfo(ip)
if want == "" {
if !errors.Is(err, ErrNoRoute) {
t.Errorf("GetIPInfo(%s) = %+v, %v; want ErrNoRoute", ip, info, err)
}
return
}
if err != nil {
t.Errorf("GetIPInfo(%s): %v", ip, err)
return
}
if info.Netblock != want || info.MaskLength != netip.MustParsePrefix(want).Bits() ||
info.ASN != origins[want] {
t.Errorf("GetIPInfo(%s) = %s (mask %d) AS%d, want %s AS%d",
ip, info.Netblock, info.MaskLength, info.ASN, want, origins[want])
}
}
lookup("10.1.2.3", "10.1.2.0/24")
lookup("::ffff:10.1.2.3", "10.1.2.0/24")
lookup("10.1.3.4", "10.1.0.0/16")
lookup("10.2.0.1", "10.0.0.0/8")
lookup("192.0.2.1", "")
lookup("2001:db8:1:2::3", "2001:db8:1:2::/64")
lookup("2001:db8:1:3::4", "2001:db8:1::/48")
lookup("2001:db8:2::1", "2001:db8::/32")
lookup("2001:db9::1", "")
err = db.DeleteLiveRouteBatch([]LiveRouteDeletion{
{
Prefix: "10.1.2.0/24", OriginASN: 64502,
PeerIP: testPeerIP, IPVersion: ipVersionV4,
},
{
Prefix: "2001:db8:1:2::/64", OriginASN: 64502,
PeerIP: testPeerIP, IPVersion: ipVersionV6,
},
})
if err != nil {
t.Fatalf("DeleteLiveRouteBatch: %v", err)
}
lookup("10.1.2.3", "10.1.0.0/16")
lookup("2001:db8:1:2::3", "2001:db8:1::/48")
}
// TestGetIPInfoIgnoresShortRoutes checks that the IP lookup does not report a
// default route, or a route shorter than /8 for IPv4 or /16 for IPv6, as the
// route for an address, and that a route of exactly /8 or /16, or longer, still
// is. The prefixes are written out so that a wrong limit fails the test.
func TestGetIPInfoIgnoresShortRoutes(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
// announce writes one live route for prefix, originated by asn.
announce := func(prefix string, asn int) {
t.Helper()
writeOriginRoutes(t, db, map[string]int{prefix: asn})
}
// lookup checks that ip resolves to the live prefix netblock from asn.
lookup := func(ip, netblock string, asn int) {
t.Helper()
info, err := db.GetIPInfo(ip)
if err != nil {
t.Errorf("GetIPInfo(%s): %v", ip, err)
return
}
if info.Netblock != netblock || info.ASN != asn {
t.Errorf("GetIPInfo(%s) = %s AS%d, want %s AS%d",
ip, info.Netblock, info.ASN, netblock, asn)
}
}
// Default and very short routes, and routes one bit shorter than each limit.
announce("2000::/3", 64500)
announce("0.0.0.0/0", 64500)
announce("2000::/15", 64500)
announce("192.0.0.0/7", 64500)
for _, ip := range []string{"2001:db8::1", "192.0.2.1"} {
info, err := db.GetIPInfo(ip)
if !errors.Is(err, ErrNoRoute) {
t.Errorf("GetIPInfo(%s) with only short routes = %+v, %v; "+
"want ErrNoRoute", ip, info, err)
}
}
// Routes of exactly the minimum length are used.
announce("2001::/16", 64501)
announce("192.0.0.0/8", 64501)
lookup("2001:db8::1", "2001::/16", 64501)
lookup("192.0.2.1", "192.0.0.0/8", 64501)
announce("2001:db8::/32", 64502)
lookup("2001:db8::1", "2001:db8::/32", 64502)
}
// writeOriginRoutes writes one live route from testPeerIP for each prefix in
// origins, originated by the AS it maps to.
func writeOriginRoutes(t *testing.T, db *Database, origins map[string]int) {
t.Helper()
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
routes := make([]*LiveRoute, 0, len(origins))
for prefix, asn := range origins {
routes = append(routes, &LiveRoute{
ID: uuid.New(),
Prefix: prefix,
MaskLength: netip.MustParsePrefix(prefix).Bits(),
IPVersion: detectIPVersion(prefix),
OriginASN: asn,
PeerIP: testPeerIP,
ASPath: []int{asn},
NextHop: testPeerIP,
LastUpdated: ts,
})
}
err := db.UpsertLiveRouteBatch(routes)
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
}
// TestConnectionPoolPragmas holds several pooled connections open at once and
// checks each one carries the per-connection settings from the DSN, plus the
// process-wide hard heap limit, and sees the new file with auto_vacuum
// incremental.
func TestConnectionPoolPragmas(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ctx := context.Background()
// Hold distinct connections open simultaneously so the pool must open a new
// one (each parsing the DSN) rather than hand back the same connection.
conns := make([]*sql.Conn, 0, heldConnections)
defer func() {
for _, c := range conns {
_ = c.Close()
}
}()
for i := range heldConnections {
c, err := db.db.Conn(ctx)
if err != nil {
t.Fatalf("failed to open connection %d: %v", i, err)
}
conns = append(conns, c)
}
for i, c := range conns {
cacheSize := readPragma(ctx, t, c, i, "cache_size")
if cacheSize != sqliteCacheSizeKiB {
t.Errorf("conn %d: cache_size = %d, want %d",
i, cacheSize, sqliteCacheSizeKiB)
}
busyTimeout := readPragma(ctx, t, c, i, "busy_timeout")
if busyTimeout != sqliteBusyTimeoutMs {
t.Errorf("conn %d: busy_timeout = %d, want %d",
i, busyTimeout, sqliteBusyTimeoutMs)
}
tempStore := readPragma(ctx, t, c, i, "temp_store")
if tempStore == tempStoreMemory {
t.Errorf("conn %d: temp_store = %d, want anything but %d (MEMORY)",
i, tempStore, tempStoreMemory)
}
hardHeapLimit := readPragma(ctx, t, c, i, "hard_heap_limit")
if hardHeapLimit != sqliteHardHeapLimitBytes {
t.Errorf("conn %d: hard_heap_limit = %d, want %d",
i, hardHeapLimit, sqliteHardHeapLimitBytes)
}
autoVacuum := readPragma(ctx, t, c, i, "auto_vacuum")
if autoVacuum != autoVacuumIncremental {
t.Errorf("conn %d: auto_vacuum = %d, want %d (incremental)",
i, autoVacuum, autoVacuumIncremental)
}
}
}
// readPragma reads the integer value of PRAGMA name on connection i, c.
func readPragma(
ctx context.Context, t *testing.T, c *sql.Conn, i int, name string,
) int64 {
t.Helper()
var value int64
err := c.QueryRowContext(ctx, "PRAGMA "+name).Scan(&value)
if err != nil {
t.Fatalf("conn %d: failed to read %s: %v", i, name, err)
}
return value
}
// TestVacuumReturnsFreePages checks that after routes are deleted from a new
// database, one Vacuum call returns every page they used. The deletes leave
// fewer free pages than the 1000 Vacuum frees per call, so none may remain.
// With auto_vacuum off (issue https://git.eeqj.de/sneak/routewatch/issues/43)
// the free pages stayed in the file, and with the PRAGMA run by ExecContext
// Vacuum freed only one page per call.
func TestVacuumReturnsFreePages(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ctx := context.Background()
ts := time.Now().UTC()
const asn = 64500
routes := make([]*LiveRoute, 0, vacuumTestRoutes)
deletions := make([]LiveRouteDeletion, 0, vacuumTestRoutes)
for i := range vacuumTestRoutes {
route := mkV4Route(t, fmt.Sprintf("10.%d.%d.0/24", i/256, i%256), asn, ts)
routes = append(routes, route)
deletions = append(deletions, LiveRouteDeletion{
Prefix: route.Prefix,
OriginASN: asn,
PeerIP: route.PeerIP,
IPVersion: ipVersionV4,
})
}
err = db.UpsertLiveRouteBatch(routes)
if err != nil {
t.Fatalf("failed to write routes: %v", err)
}
err = db.DeleteLiveRouteBatch(deletions)
if err != nil {
t.Fatalf("failed to delete routes: %v", err)
}
var before int
err = db.db.QueryRowContext(ctx, "PRAGMA freelist_count").Scan(&before)
if err != nil {
t.Fatalf("failed to read freelist_count: %v", err)
}
if before == 0 {
t.Fatalf("no free pages after deleting %d routes", vacuumTestRoutes)
}
err = db.Vacuum(ctx)
if err != nil {
t.Fatalf("Vacuum failed: %v", err)
}
var after int
err = db.db.QueryRowContext(ctx, "PRAGMA freelist_count").Scan(&after)
if err != nil {
t.Fatalf("failed to read freelist_count: %v", err)
}
if after != 0 {
t.Errorf("free pages after Vacuum = %d of %d, want 0", after, before)
}
}
// TestBatchWriteDuringCheckpoint reproduces issue #25. A batch write reads
// (SELECT) before it writes (INSERT/UPDATE). Under the default deferred locking
// the transaction begins as a reader and, when the maintainer's WAL checkpoint
// holds the write lock, its upgrade to writer fails immediately with "database
// is locked" without honouring busy_timeout, dropping the batch. With
// _txlock=immediate the transaction takes the write lock at BEGIN and waits, so
// no batch is dropped. The checkpoint runs without the Database mutex, exactly
// as the background maintainer does in production.
func TestBatchWriteDuringCheckpoint(t *testing.T) {
t.Parallel()
cfg := &config.Config{StateDir: t.TempDir()}
db, err := New(cfg, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
ctx, cancel := context.WithCancel(context.Background())
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
for {
select {
case <-ctx.Done():
return
default:
_ = db.Checkpoint(ctx) // errors are the checkpoint's own to absorb
}
}
}()
ts := time.Now().UTC()
for i := range contentionIterations {
asns := map[int]time.Time{
i % contendedASNCount: ts,
(i % contendedASNCount) + asnSecondBand: ts,
}
err := db.GetOrCreateASNBatch(asns)
if err != nil {
cancel()
wg.Wait()
t.Fatalf("batch write failed under checkpoint contention: %v", err)
}
}
cancel()
wg.Wait()
}
internal/database/interface.go
+64 -3
View File
@@ -1,41 +1,102 @@
// Package database provides SQLite storage for BGP routing data including ASNs,
// prefixes, announcements, peerings, and live route tables.
package database
import (
"context"
"time"
)
// Stats contains database statistics
// Stats contains database statistics including counts for ASNs, prefixes,
// peerings, peers, and live routes, as well as file size and prefix distribution data.
type Stats struct {
ASNs int
Prefixes int
IPv4Prefixes int
IPv6Prefixes int
Peerings int
Peers int
FileSizeBytes int64
LiveRoutes int
IPv4Routes int
IPv6Routes int
OldestRoute *time.Time
NewestRoute *time.Time
IPv4PrefixDistribution []PrefixDistribution
IPv6PrefixDistribution []PrefixDistribution
}
// Store defines the interface for database operations
// Store defines the interface for database operations. It provides methods for
// managing ASNs, prefixes, announcements, peerings, BGP peers, and live routes.
// Implementations must be safe for concurrent use.
//
//nolint:interfacebloat // the one seam handlers and tests depend on
type Store interface {
// ASN operations
GetOrCreateASN(number int, timestamp time.Time) (*ASN, error)
GetOrCreateASNBatch(asns map[int]time.Time) error
// Prefix operations
GetOrCreatePrefix(prefix string, timestamp time.Time) (*Prefix, error)
UpdatePrefixesBatch(prefixes map[string]time.Time) error
// Announcement operations
RecordAnnouncement(announcement *Announcement) error
// Peering operations
RecordPeering(fromASNID, toASNID string, timestamp time.Time) error
RecordPeering(asA, asB int, timestamp time.Time) error
// Statistics
GetStats() (Stats, error)
GetStatsContext(ctx context.Context) (Stats, error)
// Peer operations
UpdatePeer(peerIP string, peerASN int, messageType string, timestamp time.Time) error
UpdatePeerBatch(peers map[string]PeerUpdate) error
// Live route operations
UpsertLiveRoute(route *LiveRoute) error
UpsertLiveRouteBatch(routes []*LiveRoute) error
DeleteLiveRoute(prefix string, originASN int, peerIP string) error
DeleteLiveRouteBatch(deletions []LiveRouteDeletion) error
GetPrefixDistribution() (
ipv4 []PrefixDistribution, ipv6 []PrefixDistribution, err error)
GetPrefixDistributionContext(ctx context.Context) (
ipv4 []PrefixDistribution, ipv6 []PrefixDistribution, err error)
GetLiveRouteCounts() (ipv4Count, ipv6Count int, err error)
GetLiveRouteCountsContext(ctx context.Context) (ipv4Count, ipv6Count int, err error)
// IP lookup operations
GetIPInfo(ip string) (*IPInfo, error)
GetIPInfoContext(ctx context.Context, ip string) (*IPInfo, error)
// ASN WHOIS operations
GetNextStaleASN(ctx context.Context, staleThreshold time.Duration) (int, error)
UpdateASNWHOIS(ctx context.Context, update *ASNWHOISUpdate) error
GetWHOISStats(ctx context.Context, staleThreshold time.Duration) (*WHOISStats, error)
// AS and prefix detail operations
GetASDetails(asn int) (*ASN, []LiveRoute, error)
GetASDetailsContext(ctx context.Context, asn int) (*ASN, []LiveRoute, error)
GetASPeers(asn int) ([]ASPeer, error)
GetASPeersContext(ctx context.Context, asn int) ([]ASPeer, error)
GetPrefixDetails(prefix string) ([]LiveRoute, map[int]ASN, error)
GetPrefixDetailsContext(
ctx context.Context, prefix string,
) ([]LiveRoute, map[int]ASN, error)
GetRandomPrefixesByLength(maskLength, ipVersion, limit int) ([]LiveRoute, error)
GetRandomPrefixesByLengthContext(
ctx context.Context, maskLength, ipVersion, limit int,
) ([]LiveRoute, error)
// Lifecycle
Close() error
// Maintenance operations
Vacuum(ctx context.Context) error
Analyze(ctx context.Context) error
Checkpoint(ctx context.Context) error
Ping(ctx context.Context) error
}
// Ensure Database implements Store
internal/database/models.go
+129 -23
View File
@@ -6,42 +6,148 @@ import (
"github.com/google/uuid"
)
// ASN represents an Autonomous System Number
// ASN represents an Autonomous System Number with its metadata including
// handle, description, WHOIS data, and first/last seen timestamps.
//
//nolint:tagliatelle // field names are the existing JSON API
type ASN struct {
ID uuid.UUID `json:"id"`
Number int `json:"number"`
ASN int `json:"asn"`
Handle string `json:"handle"`
Description string `json:"description"`
// WHOIS parsed fields
ASName string `json:"as_name,omitempty"`
OrgName string `json:"org_name,omitempty"`
OrgID string `json:"org_id,omitempty"`
Address string `json:"address,omitempty"`
CountryCode string `json:"country_code,omitempty"`
AbuseEmail string `json:"abuse_email,omitempty"`
AbusePhone string `json:"abuse_phone,omitempty"`
TechEmail string `json:"tech_email,omitempty"`
TechPhone string `json:"tech_phone,omitempty"`
RIR string `json:"rir,omitempty"` // ARIN, RIPE, APNIC, LACNIC, AFRINIC
RIRRegDate *time.Time `json:"rir_registration_date,omitempty"`
RIRLastMod *time.Time `json:"rir_last_modified,omitempty"`
WHOISRaw string `json:"whois_raw,omitempty"`
// Timestamps
FirstSeen time.Time `json:"first_seen"`
LastSeen time.Time `json:"last_seen"`
WHOISUpdatedAt *time.Time `json:"whois_updated_at,omitempty"`
}
// Prefix represents an IP prefix (CIDR block)
// Prefix represents an IP prefix (CIDR block) with its IP version (4 or 6)
// and first/last seen timestamps.
type Prefix struct {
ID uuid.UUID
Prefix string
IPVersion int // 4 or 6
FirstSeen time.Time
LastSeen time.Time
}
// Announcement represents a BGP announcement or withdrawal event,
// containing the prefix, AS path, origin ASN, peer ASN, next hop, and timestamp.
type Announcement struct {
ID uuid.UUID
PrefixID uuid.UUID
PeerASN int
OriginASN int
Path string // JSON-encoded AS path
NextHop string
Timestamp time.Time
IsWithdrawal bool
}
// ASNPeering represents a peering relationship between two ASNs,
// stored with the lower ASN as ASA and the higher as ASB.
type ASNPeering struct {
ID uuid.UUID
ASA int
ASB int
FirstSeen time.Time
LastSeen time.Time
}
// LiveRoute represents a route in the live routing table
//
//nolint:tagliatelle // field names are the existing JSON API
type LiveRoute struct {
ID uuid.UUID `json:"id"`
Prefix string `json:"prefix"`
IPVersion int `json:"ip_version"` // 4 or 6
FirstSeen time.Time `json:"first_seen"`
LastSeen time.Time `json:"last_seen"`
}
// Announcement represents a BGP announcement
type Announcement struct {
ID uuid.UUID `json:"id"`
PrefixID uuid.UUID `json:"prefix_id"`
ASNID uuid.UUID `json:"asn_id"`
OriginASNID uuid.UUID `json:"origin_asn_id"`
Path string `json:"path"` // JSON-encoded AS path
MaskLength int `json:"mask_length"`
IPVersion int `json:"ip_version"`
OriginASN int `json:"origin_asn"`
PeerIP string `json:"peer_ip"`
ASPath []int `json:"as_path"`
NextHop string `json:"next_hop"`
Timestamp time.Time `json:"timestamp"`
IsWithdrawal bool `json:"is_withdrawal"`
LastUpdated time.Time `json:"last_updated"`
}
// ASNPeering represents a peering relationship between two ASNs
type ASNPeering struct {
ID uuid.UUID `json:"id"`
FromASNID uuid.UUID `json:"from_asn_id"`
ToASNID uuid.UUID `json:"to_asn_id"`
// PrefixDistribution represents the distribution of prefixes by mask length
//
//nolint:tagliatelle // field names are the existing JSON API
type PrefixDistribution struct {
MaskLength int `json:"mask_length"`
Count int `json:"count"`
}
// IPInfo represents comprehensive IP information for the /ip endpoint
//
//nolint:tagliatelle // field names are the existing JSON API
type IPInfo struct {
IP string `json:"ip"`
PTR []string `json:"ptr,omitempty"`
Netblock string `json:"netblock"`
MaskLength int `json:"mask_length"`
IPVersion int `json:"ip_version"`
NumPeers int `json:"num_peers"`
// AS information
ASN int `json:"asn"`
ASName string `json:"as_name,omitempty"`
Handle string `json:"handle,omitempty"`
Description string `json:"description,omitempty"`
OrgName string `json:"org_name,omitempty"`
OrgID string `json:"org_id,omitempty"`
Address string `json:"address,omitempty"`
CountryCode string `json:"country_code,omitempty"`
AbuseEmail string `json:"abuse_email,omitempty"`
RIR string `json:"rir,omitempty"`
// Timestamps
FirstSeen time.Time `json:"first_seen"`
LastSeen time.Time `json:"last_seen"`
// Indicates if WHOIS data needs refresh (not serialized)
NeedsWHOISRefresh bool `json:"-"`
}
// LiveRouteDeletion represents parameters for deleting a live route
type LiveRouteDeletion struct {
Prefix string
OriginASN int
PeerIP string
IPVersion int
}
// PeerUpdate represents parameters for updating a peer
type PeerUpdate struct {
PeerIP string
PeerASN int
MessageType string
Timestamp time.Time
}
// ASNWHOISUpdate contains WHOIS data for updating an ASN record.
type ASNWHOISUpdate struct {
ASN int
ASName string
OrgName string
OrgID string
Address string
CountryCode string
AbuseEmail string
AbusePhone string
TechEmail string
TechPhone string
RIR string
RIRRegDate *time.Time
RIRLastMod *time.Time
WHOISRaw string
}
internal/database/schema.sql
+94 -25
View File
@@ -1,16 +1,44 @@
-- IMPORTANT: This is the ONLY place where schema changes should be made.
-- We do NOT support migrations. All schema changes MUST be in this file.
-- DO NOT make schema changes anywhere else in the codebase.
CREATE TABLE IF NOT EXISTS asns (
id TEXT PRIMARY KEY,
number INTEGER UNIQUE NOT NULL,
asn INTEGER PRIMARY KEY,
handle TEXT,
description TEXT,
-- WHOIS parsed fields
as_name TEXT,
org_name TEXT,
org_id TEXT,
address TEXT, -- full address (may be multi-line)
country_code TEXT,
abuse_email TEXT,
abuse_phone TEXT,
tech_email TEXT,
tech_phone TEXT,
rir TEXT, -- ARIN, RIPE, APNIC, LACNIC, AFRINIC
rir_registration_date DATETIME,
rir_last_modified DATETIME,
-- Raw WHOIS response
whois_raw TEXT, -- complete WHOIS response text
-- Timestamps
first_seen DATETIME NOT NULL,
last_seen DATETIME NOT NULL,
whois_updated_at DATETIME -- when we last fetched WHOIS data
);
-- IPv4 prefixes table
CREATE TABLE IF NOT EXISTS prefixes_v4 (
id TEXT PRIMARY KEY,
prefix TEXT UNIQUE NOT NULL,
first_seen DATETIME NOT NULL,
last_seen DATETIME NOT NULL
);
CREATE TABLE IF NOT EXISTS prefixes (
-- IPv6 prefixes table
CREATE TABLE IF NOT EXISTS prefixes_v6 (
id TEXT PRIMARY KEY,
prefix TEXT UNIQUE NOT NULL,
ip_version INTEGER NOT NULL, -- 4 for IPv4, 6 for IPv6
first_seen DATETIME NOT NULL,
last_seen DATETIME NOT NULL
);
@@ -18,26 +46,23 @@ CREATE TABLE IF NOT EXISTS prefixes (
CREATE TABLE IF NOT EXISTS announcements (
id TEXT PRIMARY KEY,
prefix_id TEXT NOT NULL,
asn_id TEXT NOT NULL,
origin_asn_id TEXT NOT NULL,
peer_asn INTEGER NOT NULL,
origin_asn INTEGER NOT NULL,
path TEXT NOT NULL,
next_hop TEXT,
timestamp DATETIME NOT NULL,
is_withdrawal BOOLEAN NOT NULL DEFAULT 0,
FOREIGN KEY (prefix_id) REFERENCES prefixes(id),
FOREIGN KEY (asn_id) REFERENCES asns(id),
FOREIGN KEY (origin_asn_id) REFERENCES asns(id)
FOREIGN KEY (peer_asn) REFERENCES asns(asn),
FOREIGN KEY (origin_asn) REFERENCES asns(asn)
);
CREATE TABLE IF NOT EXISTS asn_peerings (
CREATE TABLE IF NOT EXISTS peerings (
id TEXT PRIMARY KEY,
from_asn_id TEXT NOT NULL,
to_asn_id TEXT NOT NULL,
as_a INTEGER NOT NULL,
as_b INTEGER NOT NULL,
first_seen DATETIME NOT NULL,
last_seen DATETIME NOT NULL,
FOREIGN KEY (from_asn_id) REFERENCES asns(id),
FOREIGN KEY (to_asn_id) REFERENCES asns(id),
UNIQUE(from_asn_id, to_asn_id)
UNIQUE(as_a, as_b)
);
-- BGP peers that send us messages
@@ -50,22 +75,66 @@ CREATE TABLE IF NOT EXISTS bgp_peers (
last_message_type TEXT
);
CREATE INDEX IF NOT EXISTS idx_prefixes_ip_version ON prefixes(ip_version);
CREATE INDEX IF NOT EXISTS idx_prefixes_version_prefix ON prefixes(ip_version, prefix);
-- Indexes for prefixes_v4 table
CREATE INDEX IF NOT EXISTS idx_prefixes_v4_prefix ON prefixes_v4(prefix);
-- Indexes for prefixes_v6 table
CREATE INDEX IF NOT EXISTS idx_prefixes_v6_prefix ON prefixes_v6(prefix);
CREATE INDEX IF NOT EXISTS idx_announcements_timestamp ON announcements(timestamp);
CREATE INDEX IF NOT EXISTS idx_announcements_prefix_id ON announcements(prefix_id);
CREATE INDEX IF NOT EXISTS idx_announcements_asn_id ON announcements(asn_id);
CREATE INDEX IF NOT EXISTS idx_asn_peerings_from_asn ON asn_peerings(from_asn_id);
CREATE INDEX IF NOT EXISTS idx_asn_peerings_to_asn ON asn_peerings(to_asn_id);
CREATE INDEX IF NOT EXISTS idx_asn_peerings_lookup ON asn_peerings(from_asn_id, to_asn_id);
-- Additional indexes for prefixes table
CREATE INDEX IF NOT EXISTS idx_prefixes_prefix ON prefixes(prefix);
CREATE INDEX IF NOT EXISTS idx_announcements_peer_asn ON announcements(peer_asn);
CREATE INDEX IF NOT EXISTS idx_announcements_origin_asn ON announcements(origin_asn);
CREATE INDEX IF NOT EXISTS idx_peerings_as_a ON peerings(as_a);
CREATE INDEX IF NOT EXISTS idx_peerings_as_b ON peerings(as_b);
CREATE INDEX IF NOT EXISTS idx_peerings_lookup ON peerings(as_a, as_b);
-- Indexes for asns table
CREATE INDEX IF NOT EXISTS idx_asns_number ON asns(number);
CREATE INDEX IF NOT EXISTS idx_asns_asn ON asns(asn);
CREATE INDEX IF NOT EXISTS idx_asns_whois_updated_at ON asns(whois_updated_at);
-- Indexes for bgp_peers table
CREATE INDEX IF NOT EXISTS idx_bgp_peers_asn ON bgp_peers(peer_asn);
CREATE INDEX IF NOT EXISTS idx_bgp_peers_last_seen ON bgp_peers(last_seen);
CREATE INDEX IF NOT EXISTS idx_bgp_peers_ip ON bgp_peers(peer_ip);
-- IPv4 routing table maintained by PrefixHandler
CREATE TABLE IF NOT EXISTS live_routes_v4 (
id TEXT PRIMARY KEY,
prefix TEXT NOT NULL,
mask_length INTEGER NOT NULL, -- CIDR mask length (0-32)
origin_asn INTEGER NOT NULL,
peer_ip TEXT NOT NULL,
as_path TEXT NOT NULL, -- JSON array
next_hop TEXT NOT NULL,
last_updated DATETIME NOT NULL,
UNIQUE(prefix, origin_asn, peer_ip)
);
-- IPv6 routing table maintained by PrefixHandler
CREATE TABLE IF NOT EXISTS live_routes_v6 (
id TEXT PRIMARY KEY,
prefix TEXT NOT NULL,
mask_length INTEGER NOT NULL, -- CIDR mask length (0-128)
origin_asn INTEGER NOT NULL,
peer_ip TEXT NOT NULL,
as_path TEXT NOT NULL, -- JSON array
next_hop TEXT NOT NULL,
last_updated DATETIME NOT NULL,
UNIQUE(prefix, origin_asn, peer_ip)
);
-- Indexes for live_routes_v4 table
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_prefix ON live_routes_v4(prefix);
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_mask_length ON live_routes_v4(mask_length);
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_origin_asn ON live_routes_v4(origin_asn);
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_last_updated ON live_routes_v4(last_updated);
-- Index to optimize prefix distribution queries
CREATE INDEX IF NOT EXISTS idx_live_routes_v4_mask_prefix ON live_routes_v4(mask_length, prefix);
-- Indexes for live_routes_v6 table
CREATE INDEX IF NOT EXISTS idx_live_routes_v6_prefix ON live_routes_v6(prefix);
CREATE INDEX IF NOT EXISTS idx_live_routes_v6_mask_length ON live_routes_v6(mask_length);
CREATE INDEX IF NOT EXISTS idx_live_routes_v6_origin_asn ON live_routes_v6(origin_asn);
CREATE INDEX IF NOT EXISTS idx_live_routes_v6_last_updated ON live_routes_v6(last_updated);
-- Index to optimize prefix distribution queries
CREATE INDEX IF NOT EXISTS idx_live_routes_v6_mask_prefix ON live_routes_v6(mask_length, prefix);
internal/database/slowquery.go
+22 -39
View File
@@ -3,14 +3,15 @@ package database
import (
"context"
"database/sql"
"log/slog"
"time"
"git.eeqj.de/sneak/routewatch/internal/logger"
)
const slowQueryThreshold = 50 * time.Millisecond
const slowQueryThreshold = 25 * time.Millisecond
// logSlowQuery logs queries that take longer than slowQueryThreshold
func logSlowQuery(logger *slog.Logger, query string, start time.Time) {
func logSlowQuery(logger *logger.Logger, query string, start time.Time) {
elapsed := time.Since(start)
if elapsed > slowQueryThreshold {
logger.Debug("Slow query", "query", query, "duration", elapsed)
@@ -18,28 +19,21 @@ func logSlowQuery(logger *slog.Logger, query string, start time.Time) {
}
// queryRow wraps QueryRow with slow query logging
func (d *Database) queryRow(query string, args ...interface{}) *sql.Row {
//
//nolint:unused // kept for consistency with other query wrappers
func (d *Database) queryRow(query string, args ...any) *sql.Row {
start := time.Now()
defer logSlowQuery(d.logger, query, start)
return d.db.QueryRow(query, args...)
}
// query wraps Query with slow query logging
// nolint:unused // kept for future use to ensure all queries go through slow query logging
func (d *Database) query(query string, args ...interface{}) (*sql.Rows, error) {
start := time.Now()
defer logSlowQuery(d.logger, query, start)
return d.db.Query(query, args...)
return d.db.QueryRowContext(context.Background(), query, args...)
}
// exec wraps Exec with slow query logging
func (d *Database) exec(query string, args ...interface{}) error {
func (d *Database) exec(query string, args ...any) error {
start := time.Now()
defer logSlowQuery(d.logger, query, start)
_, err := d.db.Exec(query, args...)
_, err := d.db.ExecContext(context.Background(), query, args...)
return err
}
@@ -47,35 +41,22 @@ func (d *Database) exec(query string, args ...interface{}) error {
// loggingTx wraps sql.Tx to log slow queries
type loggingTx struct {
*sql.Tx
logger *slog.Logger
logger *logger.Logger
}
// QueryRow wraps sql.Tx.QueryRow to log slow queries
func (tx *loggingTx) QueryRow(query string, args ...interface{}) *sql.Row {
func (tx *loggingTx) QueryRow(query string, args ...any) *sql.Row {
start := time.Now()
defer logSlowQuery(tx.logger, query, start)
return tx.Tx.QueryRow(query, args...)
}
// Query wraps sql.Tx.Query to log slow queries
func (tx *loggingTx) Query(query string, args ...interface{}) (*sql.Rows, error) {
start := time.Now()
defer logSlowQuery(tx.logger, query, start)
return tx.Tx.Query(query, args...)
}
// QueryContext wraps sql.Tx.QueryContext to log slow queries
func (tx *loggingTx) QueryContext(ctx context.Context, query string, args ...interface{}) (*sql.Rows, error) {
start := time.Now()
defer logSlowQuery(tx.logger, query, start)
return tx.Tx.QueryContext(ctx, query, args...)
return tx.Tx.QueryRowContext(context.Background(), query, args...)
}
// QueryRowContext wraps sql.Tx.QueryRowContext to log slow queries
func (tx *loggingTx) QueryRowContext(ctx context.Context, query string, args ...interface{}) *sql.Row {
func (tx *loggingTx) QueryRowContext(
ctx context.Context, query string, args ...any,
) *sql.Row {
start := time.Now()
defer logSlowQuery(tx.logger, query, start)
@@ -83,15 +64,17 @@ func (tx *loggingTx) QueryRowContext(ctx context.Context, query string, args ...
}
// Exec wraps sql.Tx.Exec to log slow queries
func (tx *loggingTx) Exec(query string, args ...interface{}) (sql.Result, error) {
func (tx *loggingTx) Exec(query string, args ...any) (sql.Result, error) {
start := time.Now()
defer logSlowQuery(tx.logger, query, start)
return tx.Tx.Exec(query, args...)
return tx.Tx.ExecContext(context.Background(), query, args...)
}
// ExecContext wraps sql.Tx.ExecContext to log slow queries
func (tx *loggingTx) ExecContext(ctx context.Context, query string, args ...interface{}) (sql.Result, error) {
func (tx *loggingTx) ExecContext(
ctx context.Context, query string, args ...any,
) (sql.Result, error) {
start := time.Now()
defer logSlowQuery(tx.logger, query, start)
internal/database/utils.go
+14 -5
View File
@@ -1,6 +1,8 @@
package database
import (
"fmt"
"net"
"strings"
"github.com/google/uuid"
@@ -10,11 +12,6 @@ func generateUUID() uuid.UUID {
return uuid.New()
}
const (
ipVersionV4 = 4
ipVersionV6 = 6
)
// detectIPVersion determines if a prefix is IPv4 (returns 4) or IPv6 (returns 6)
func detectIPVersion(prefix string) int {
if strings.Contains(prefix, ":") {
@@ -23,3 +20,15 @@ func detectIPVersion(prefix string) int {
return ipVersionV4
}
// prefixMaskLength returns the mask length of a prefix such as 192.0.2.0/24.
func prefixMaskLength(prefix string) (int, error) {
_, network, err := net.ParseCIDR(prefix)
if err != nil {
return 0, fmt.Errorf("invalid prefix %s: %w", prefix, err)
}
maskLength, _ := network.Mask.Size()
return maskLength, nil
}
internal/logger/logger.go
+176
View File
@@ -0,0 +1,176 @@
// Package logger provides a structured logger with source location tracking.
// It wraps the standard library's log/slog package and automatically enriches
// log messages with the file name, line number, and function name of the caller.
// The output format is automatically selected based on the runtime environment:
// human-readable text for terminals, JSON for non-terminal output.
package logger
import (
"fmt"
"log/slog"
"os"
"path/filepath"
"runtime"
"strings"
"golang.org/x/term"
)
// Logger wraps slog.Logger to add automatic source location information
// to all log messages. It embeds slog.Logger and provides the same logging
// methods (Debug, Info, Warn, Error) but enriches each message with the
// file name, line number, and function name of the caller.
type Logger struct {
*slog.Logger
}
// AsSlog returns the underlying slog.Logger for use with APIs that require
// a standard slog.Logger instance rather than the custom Logger type.
func (l *Logger) AsSlog() *slog.Logger {
return l.Logger
}
// New creates a new Logger with an appropriate handler based on the runtime
// environment. If stdout is a terminal, it uses a human-readable text format;
// otherwise, it outputs JSON for structured log aggregation. The log level
// defaults to Info, but can be set to Debug by including "routewatch" in the
// DEBUG environment variable.
func New() *Logger {
level := slog.LevelInfo
if debug := os.Getenv("DEBUG"); strings.Contains(debug, "routewatch") {
level = slog.LevelDebug
}
opts := &slog.HandlerOptions{
Level: level,
}
var handler slog.Handler
if term.IsTerminal(int(os.Stdout.Fd())) {
// Terminal, use text
handler = slog.NewTextHandler(os.Stdout, opts)
} else {
// Not a terminal, use JSON
handler = slog.NewJSONHandler(os.Stdout, opts)
}
return &Logger{Logger: slog.New(handler)}
}
// sourceSkipLevel defines the number of call stack frames to skip when
// determining the caller's source location. This accounts for the logger
// method itself and the getSourceAttrs helper function.
const sourceSkipLevel = 2
// getSourceAttrs returns attributes for the calling source location
func getSourceAttrs() []slog.Attr {
pc, file, line, ok := runtime.Caller(sourceSkipLevel)
if !ok {
return nil
}
// Get just the filename without the full path
file = filepath.Base(file)
// Get the function name
fn := runtime.FuncForPC(pc)
var funcName string
if fn != nil {
funcName = filepath.Base(fn.Name())
}
attrs := []slog.Attr{
slog.String("source", fmt.Sprintf("%s:%d", file, line)),
}
if funcName != "" {
attrs = append(attrs, slog.String("func", funcName))
}
return attrs
}
// Debug logs a message at debug level with automatic source location tracking.
// Additional structured attributes can be passed as key-value pairs in args.
// Debug messages are only output when the DEBUG environment variable contains
// "routewatch".
func (l *Logger) Debug(msg string, args ...any) {
sourceAttrs := getSourceAttrs()
allArgs := make([]any, 0, len(args)+len(sourceAttrs)*2)
// Add source attributes first
for _, attr := range sourceAttrs {
allArgs = append(allArgs, attr)
}
// Add user args
allArgs = append(allArgs, args...)
l.Logger.Debug(msg, allArgs...)
}
// Info logs a message at info level with automatic source location tracking.
// Additional structured attributes can be passed as key-value pairs in args.
func (l *Logger) Info(msg string, args ...any) {
sourceAttrs := getSourceAttrs()
allArgs := make([]any, 0, len(args)+len(sourceAttrs)*2)
// Add source attributes first
for _, attr := range sourceAttrs {
allArgs = append(allArgs, attr)
}
// Add user args
allArgs = append(allArgs, args...)
l.Logger.Info(msg, allArgs...)
}
// Warn logs a message at warn level with automatic source location tracking.
// Additional structured attributes can be passed as key-value pairs in args.
func (l *Logger) Warn(msg string, args ...any) {
sourceAttrs := getSourceAttrs()
allArgs := make([]any, 0, len(args)+len(sourceAttrs)*2)
// Add source attributes first
for _, attr := range sourceAttrs {
allArgs = append(allArgs, attr)
}
// Add user args
allArgs = append(allArgs, args...)
l.Logger.Warn(msg, allArgs...)
}
// Error logs a message at error level with automatic source location tracking.
// Additional structured attributes can be passed as key-value pairs in args.
func (l *Logger) Error(msg string, args ...any) {
sourceAttrs := getSourceAttrs()
allArgs := make([]any, 0, len(args)+len(sourceAttrs)*2)
// Add source attributes first
for _, attr := range sourceAttrs {
allArgs = append(allArgs, attr)
}
// Add user args
allArgs = append(allArgs, args...)
l.Logger.Error(msg, allArgs...)
}
// With returns a new Logger with additional structured attributes that will
// be included in all subsequent log messages. The args parameter accepts
// key-value pairs in the same format as the logging methods.
func (l *Logger) With(args ...any) *Logger {
return &Logger{Logger: l.Logger.With(args...)}
}
// WithGroup returns a new Logger that adds the specified group name as a
// prefix to all attribute keys in subsequent log messages. This is useful
// for organizing related attributes under a common namespace.
func (l *Logger) WithGroup(name string) *Logger {
return &Logger{Logger: l.Logger.WithGroup(name)}
}
internal/metrics/metrics.go
+150 -5
View File
@@ -15,12 +15,29 @@ type Tracker struct {
registry metrics.Registry
connectedSince time.Time
isConnected atomic.Bool
reconnectCount atomic.Uint64
// Stream metrics
// Stream metrics (decompressed data)
messageCounter metrics.Counter
byteCounter metrics.Counter
messageRate metrics.Meter
byteRate metrics.Meter
// Wire bytes metrics (actual bytes on the wire, before decompression)
wireByteCounter metrics.Counter
wireByteRate metrics.Meter
// Route update metrics
ipv4UpdateRate metrics.Meter
ipv6UpdateRate metrics.Meter
// Announcement/withdrawal metrics
announcementCounter metrics.Counter
withdrawalCounter metrics.Counter
churnRate metrics.Meter // combined announcements + withdrawals per second
// BGP peer tracking
bgpPeerCount atomic.Int32
}
// New creates a new metrics tracker
@@ -33,25 +50,47 @@ func New() *Tracker {
byteCounter: metrics.NewCounter(),
messageRate: metrics.NewMeter(),
byteRate: metrics.NewMeter(),
wireByteCounter: metrics.NewCounter(),
wireByteRate: metrics.NewMeter(),
ipv4UpdateRate: metrics.NewMeter(),
ipv6UpdateRate: metrics.NewMeter(),
announcementCounter: metrics.NewCounter(),
withdrawalCounter: metrics.NewCounter(),
churnRate: metrics.NewMeter(),
}
}
// SetConnected updates the connection status
// SetConnected updates the connection status. Every connection after the
// first counts one reconnect.
func (t *Tracker) SetConnected(connected bool) {
t.isConnected.Store(connected)
if connected {
if !connected {
return
}
t.mu.Lock()
// connectedSince is zero until the first connection.
connectedBefore := !t.connectedSince.IsZero()
t.connectedSince = time.Now()
t.mu.Unlock()
if connectedBefore {
t.reconnectCount.Add(1)
}
}
// GetReconnectCount returns the number of reconnections since startup
func (t *Tracker) GetReconnectCount() uint64 {
return t.reconnectCount.Load()
}
// IsConnected returns the current connection status
func (t *Tracker) IsConnected() bool {
return t.isConnected.Load()
}
// RecordMessage records a received message and its size
// RecordMessage records a received message and its decompressed size
func (t *Tracker) RecordMessage(bytes int64) {
t.messageCounter.Inc(1)
t.byteCounter.Inc(bytes)
@@ -59,6 +98,12 @@ func (t *Tracker) RecordMessage(bytes int64) {
t.byteRate.Mark(bytes)
}
// RecordWireBytes records actual bytes received on the wire (before decompression)
func (t *Tracker) RecordWireBytes(bytes int64) {
t.wireByteCounter.Inc(bytes)
t.wireByteRate.Mark(bytes)
}
// GetStreamMetrics returns current streaming metrics
func (t *Tracker) GetStreamMetrics() StreamMetrics {
t.mu.RLock()
@@ -70,31 +115,131 @@ func (t *Tracker) GetStreamMetrics() StreamMetrics {
// Safely convert counters to uint64
msgCount := t.messageCounter.Count()
byteCount := t.byteCounter.Count()
wireByteCount := t.wireByteCounter.Count()
var totalMessages, totalBytes uint64
var totalMessages, totalBytes, totalWireBytes uint64
if msgCount >= 0 {
totalMessages = uint64(msgCount)
}
if byteCount >= 0 {
totalBytes = uint64(byteCount)
}
if wireByteCount >= 0 {
totalWireBytes = uint64(wireByteCount)
}
return StreamMetrics{
TotalMessages: totalMessages,
TotalBytes: totalBytes,
TotalWireBytes: totalWireBytes,
ConnectedSince: connectedSince,
Connected: t.isConnected.Load(),
MessagesPerSec: t.messageRate.Rate1(),
BitsPerSec: t.byteRate.Rate1() * bitsPerByte,
WireBitsPerSec: t.wireByteRate.Rate1() * bitsPerByte,
ReconnectCount: t.reconnectCount.Load(),
}
}
// RecordIPv4Update records an IPv4 route update
func (t *Tracker) RecordIPv4Update() {
t.ipv4UpdateRate.Mark(1)
}
// RecordIPv6Update records an IPv6 route update
func (t *Tracker) RecordIPv6Update() {
t.ipv6UpdateRate.Mark(1)
}
// RecordAnnouncement records a route announcement
func (t *Tracker) RecordAnnouncement() {
t.announcementCounter.Inc(1)
t.churnRate.Mark(1)
}
// RecordWithdrawal records a route withdrawal
func (t *Tracker) RecordWithdrawal() {
t.withdrawalCounter.Inc(1)
t.churnRate.Mark(1)
}
// SetBGPPeerCount updates the current BGP peer count
func (t *Tracker) SetBGPPeerCount(count int) {
// BGP peer count is always small (< 1000), so int32 is safe
if count > 0 && count < 1<<31 {
t.bgpPeerCount.Store(int32(count))
}
}
// GetBGPPeerCount returns the current BGP peer count
func (t *Tracker) GetBGPPeerCount() int {
return int(t.bgpPeerCount.Load())
}
// GetAnnouncementCount returns the total announcement count
func (t *Tracker) GetAnnouncementCount() uint64 {
count := t.announcementCounter.Count()
if count < 0 {
return 0
}
return uint64(count)
}
// GetWithdrawalCount returns the total withdrawal count
func (t *Tracker) GetWithdrawalCount() uint64 {
count := t.withdrawalCounter.Count()
if count < 0 {
return 0
}
return uint64(count)
}
// GetChurnRate returns the route churn rate per second
func (t *Tracker) GetChurnRate() float64 {
return t.churnRate.Rate1()
}
// GetRouteMetrics returns current route update metrics
func (t *Tracker) GetRouteMetrics() RouteMetrics {
return RouteMetrics{
IPv4UpdatesPerSec: t.ipv4UpdateRate.Rate1(),
IPv6UpdatesPerSec: t.ipv6UpdateRate.Rate1(),
}
}
// StreamMetrics contains streaming statistics
type StreamMetrics struct {
// TotalMessages is the total number of messages received since startup
TotalMessages uint64
// TotalBytes is the total number of decompressed bytes received since startup
TotalBytes uint64
// TotalWireBytes is the total number of bytes received on the wire (before
// decompression)
TotalWireBytes uint64
// ConnectedSince is the time when the current connection was established
ConnectedSince time.Time
// Connected indicates whether the stream is currently connected
Connected bool
// MessagesPerSec is the rate of messages received per second (1-minute average)
MessagesPerSec float64
// BitsPerSec is the rate of decompressed bits received per second (1-minute
// average)
BitsPerSec float64
// WireBitsPerSec is the rate of bits received on the wire per second
// (1-minute average)
WireBitsPerSec float64
// ReconnectCount is the number of reconnections since startup
ReconnectCount uint64
}
// RouteMetrics contains route update statistics
type RouteMetrics struct {
// IPv4UpdatesPerSec is the rate of IPv4 route updates per second (1-minute average)
IPv4UpdatesPerSec float64
// IPv6UpdatesPerSec is the rate of IPv6 route updates per second (1-minute average)
IPv6UpdatesPerSec float64
}
internal/metrics/metrics_test.go
+27
View File
@@ -0,0 +1,27 @@
package metrics_test
import (
"testing"
"git.eeqj.de/sneak/routewatch/internal/metrics"
)
// TestReconnectCount connects three times with a disconnect between each, the
// way the streamer does, and expects every connection after the first to count
// one reconnect.
func TestReconnectCount(t *testing.T) {
t.Parallel()
tracker := metrics.New()
tracker.SetConnected(true)
tracker.SetConnected(false)
tracker.SetConnected(true)
tracker.SetConnected(false)
tracker.SetConnected(true)
const wantReconnects = 2
if got := tracker.GetReconnectCount(); got != wantReconnects {
t.Fatalf("reconnect count = %d, want %d", got, wantReconnects)
}
}
internal/ristypes/ris.go
+37 -12
View File
@@ -1,4 +1,5 @@
// Package ristypes defines the data structures for RIS Live BGP messages and announcements.
// Package ristypes defines the data structures for RIS Live BGP messages and
// announcements.
package ristypes
import (
@@ -6,32 +7,41 @@ import (
"time"
)
// ASPath represents an AS path that may contain nested AS sets
// ASPath represents a BGP AS path as a slice of AS numbers.
// It handles JSON unmarshaling of both simple arrays and nested AS sets,
// flattening any nested structures into a single sequence of AS numbers.
type ASPath []int
// UnmarshalJSON implements custom JSON unmarshaling to flatten nested arrays
// UnmarshalJSON implements the json.Unmarshaler interface for ASPath.
// It handles both simple integer arrays [1, 2, 3] and nested AS sets
// like [1, [2, 3], 4], flattening them into a single slice of integers.
func (p *ASPath) UnmarshalJSON(data []byte) error {
// First try to unmarshal as a simple array of integers
var simple []int
if err := json.Unmarshal(data, &simple); err == nil {
err := json.Unmarshal(data, &simple)
if err == nil {
*p = ASPath(simple)
return nil
}
// If that fails, unmarshal as array of interfaces and flatten
var raw []interface{}
if err := json.Unmarshal(data, &raw); err != nil {
var raw []any
err = json.Unmarshal(data, &raw)
if err != nil {
return err
}
// Flatten the array
result := make([]int, 0)
for _, item := range raw {
switch v := item.(type) {
case float64:
result = append(result, int(v))
case []interface{}:
case []any:
// Nested array - flatten it
for _, nested := range v {
if num, ok := nested.(float64); ok {
@@ -46,13 +56,20 @@ func (p *ASPath) UnmarshalJSON(data []byte) error {
return nil
}
// RISLiveMessage represents the outer wrapper from the RIS Live stream
// RISLiveMessage represents the outer wrapper message from the RIPE RIS Live stream.
// Each message contains a Type field indicating the message type and a Data field
// containing the actual BGP message payload.
type RISLiveMessage struct {
Type string `json:"type"`
Data RISMessage `json:"data"`
}
// RISMessage represents a message from the RIS Live stream
// RISMessage represents a BGP update message from the RIPE RIS Live stream.
// It contains metadata about the BGP session (peer, ASN, host) along with
// the actual BGP update data including AS path, communities, announcements,
// and withdrawals.
//
//nolint:tagliatelle // field names are the RIS Live feed's
type RISMessage struct {
Type string `json:"type"`
Timestamp float64 `json:"timestamp"`
@@ -65,16 +82,24 @@ type RISMessage struct {
MrtTime float64 `json:"mrt_time,omitempty"`
SocketTime float64 `json:"socket_time,omitempty"`
Path ASPath `json:"path,omitempty"`
Community [][]int `json:"community,omitempty"`
// Community and Raw are present in the feed but read by no handler.
// They are the largest fields on a message that lives in up to four
// handler queues, so json:"-" keeps them out of the decoded message
// to save queue memory. Do not decode them without a consumer.
Community [][]int `json:"-"`
Origin string `json:"origin,omitempty"`
MED *int `json:"med,omitempty"`
LocalPref *int `json:"local_pref,omitempty"`
Announcements []RISAnnouncement `json:"announcements,omitempty"`
Withdrawals []string `json:"withdrawals,omitempty"`
Raw string `json:"raw,omitempty"`
Raw string `json:"-"`
}
// RISAnnouncement represents announcement data within a RIS message
// RISAnnouncement represents a BGP route announcement within a RIS message.
// It contains the next hop IP address and the list of prefixes being announced
// via that next hop.
//
//nolint:tagliatelle // field names are the RIS Live feed's
type RISAnnouncement struct {
NextHop string `json:"next_hop"`
Prefixes []string `json:"prefixes"`
internal/ristypes/ris_test.go
+84
View File
@@ -0,0 +1,84 @@
package ristypes_test
import (
"bufio"
"encoding/json"
"os"
"testing"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
)
// messageExamplesPath is the captured RIS Live feed used as decode fixtures,
// one JSON message per line.
const messageExamplesPath = "../../docs/message-examples.json"
// TestDecodeDropsCommunityAndRaw decodes every captured message the way the
// streamer does and checks that the fields no handler reads (Community, Raw)
// stay empty while the fields handlers use (Path, Announcements) still decode.
func TestDecodeDropsCommunityAndRaw(t *testing.T) {
t.Parallel()
f, err := os.Open(messageExamplesPath)
if err != nil {
t.Fatalf("open fixtures: %v", err)
}
defer func() {
_ = f.Close()
}()
var messages, withPath, withAnnouncements int
scanner := bufio.NewScanner(f)
scanner.Buffer(make([]byte, 0, 64*1024), 1024*1024)
for scanner.Scan() {
line := scanner.Bytes()
if len(line) == 0 {
continue
}
var wrapper ristypes.RISLiveMessage
err := json.Unmarshal(line, &wrapper)
if err != nil {
t.Fatalf("unmarshal message %d: %v", messages+1, err)
}
messages++
msg := wrapper.Data
if msg.Community != nil {
t.Errorf("message %d: Community decoded, want empty: %v",
messages, msg.Community)
}
if msg.Raw != "" {
t.Errorf("message %d: Raw decoded, want empty", messages)
}
if len(msg.Path) > 0 {
withPath++
}
if len(msg.Announcements) > 0 {
withAnnouncements++
}
}
err = scanner.Err()
if err != nil {
t.Fatalf("scan fixtures: %v", err)
}
if messages == 0 {
t.Fatal("no messages decoded from fixtures")
}
// The fixtures include announcement messages; a used field must still decode,
// otherwise an empty Community/Raw would prove nothing.
if withPath == 0 || withAnnouncements == 0 {
t.Fatalf("used fields did not decode: withPath=%d withAnnouncements=%d",
withPath, withAnnouncements)
}
}
internal/routewatch/app.go
+98 -125
View File
@@ -1,88 +1,66 @@
// Package routewatch contains the primary RouteWatch type that represents a running instance
// of the application and contains pointers to its core dependencies, and is responsible for initialization.
// Package routewatch contains the primary RouteWatch type that represents a
// running instance of the application and contains pointers to its core
// dependencies, and is responsible for initialization.
package routewatch
import (
"context"
"fmt"
"log/slog"
"os"
"strings"
"errors"
"sync"
"time"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/routingtable"
"git.eeqj.de/sneak/routewatch/internal/server"
"git.eeqj.de/sneak/routewatch/internal/snapshotter"
"git.eeqj.de/sneak/routewatch/internal/streamer"
"go.uber.org/fx"
)
const (
// routingTableStatsInterval is how often we log routing table statistics
routingTableStatsInterval = 15 * time.Second
)
// errNonBatchedNotImplemented is returned by Run when batched database writes
// are disabled, since only the batched handlers exist.
var errNonBatchedNotImplemented = errors.New("non-batched handlers not implemented")
// Dependencies contains all dependencies for RouteWatch
type Dependencies struct {
fx.In
DB database.Store
RoutingTable *routingtable.RoutingTable
Streamer *streamer.Streamer
Server *server.Server
Logger *slog.Logger
Logger *logger.Logger
Config *config.Config
}
// RouteWatch represents the main application instance
type RouteWatch struct {
db database.Store
routingTable *routingtable.RoutingTable
streamer *streamer.Streamer
server *server.Server
snapshotter *snapshotter.Snapshotter
logger *slog.Logger
logger *logger.Logger
maxRuntime time.Duration
shutdown bool
mu sync.Mutex
}
// isTruthy returns true if the value is considered truthy
// Empty string, "0", and "false" are considered falsy, everything else is truthy
func isTruthy(value string) bool {
return value != "" && value != "0" && value != "false"
}
// isSnapshotterEnabled checks if the snapshotter should be enabled based on environment variable
func isSnapshotterEnabled() bool {
return !isTruthy(os.Getenv("ROUTEWATCH_DISABLE_SNAPSHOTTER"))
config *config.Config
dbHandler *ASHandler
peerHandler *PeerHandler
prefixHandler *PrefixHandler
peeringHandler *PeeringHandler
asnFetcher *ASNFetcher
dbMaintainer *DBMaintainer
}
// New creates a new RouteWatch instance
func New(deps Dependencies) *RouteWatch {
rw := &RouteWatch{
db: deps.DB,
routingTable: deps.RoutingTable,
streamer: deps.Streamer,
server: deps.Server,
logger: deps.Logger,
maxRuntime: deps.Config.MaxRuntime,
}
// Create snapshotter if enabled
if isSnapshotterEnabled() {
snap, err := snapshotter.New(deps.RoutingTable, deps.Config, deps.Logger)
if err != nil {
deps.Logger.Error("Failed to create snapshotter", "error", err)
// Continue without snapshotter
} else {
rw.snapshotter = snap
}
config: deps.Config,
}
return rw
@@ -95,41 +73,62 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
// Apply runtime limit if specified
if rw.maxRuntime > 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, rw.maxRuntime)
defer cancel()
rw.logger.Info("Running with time limit", "max_runtime", rw.maxRuntime)
}
// Register database handler to process BGP UPDATE messages
dbHandler := NewDatabaseHandler(rw.db, rw.logger)
rw.streamer.RegisterHandler(dbHandler)
if rw.config.EnableBatchedDatabaseWrites {
rw.logger.Info("Using batched database handlers for improved performance")
// ASHandler maintains the asns table
rw.dbHandler = NewASHandler(rw.db, rw.logger)
rw.streamer.RegisterHandler(rw.dbHandler)
// Register routing table handler to maintain in-memory routing table
rtHandler := NewRoutingTableHandler(rw.routingTable, rw.logger)
rw.streamer.RegisterHandler(rtHandler)
// PeerHandler maintains the bgp_peers table
rw.peerHandler = NewPeerHandler(rw.db, rw.logger)
rw.streamer.RegisterHandler(rw.peerHandler)
// Register peer tracking handler to track all peers
peerHandler := NewPeerHandler(rw.db, rw.logger)
rw.streamer.RegisterHandler(peerHandler)
// PrefixHandler maintains the prefixes and live_routes tables
rw.prefixHandler = NewPrefixHandler(rw.db, rw.logger)
rw.prefixHandler.SetMetricsTracker(rw.streamer.GetMetricsTracker())
rw.streamer.RegisterHandler(rw.prefixHandler)
// Start periodic routing table stats logging
go rw.logRoutingTableStats(ctx)
// PeeringHandler maintains the asn_peerings table
rw.peeringHandler = NewPeeringHandler(rw.db, rw.logger)
rw.streamer.RegisterHandler(rw.peeringHandler)
} else {
// Non-batched handlers not implemented yet
rw.logger.Error("Non-batched handlers not implemented")
// Start snapshotter if available
if rw.snapshotter != nil {
rw.snapshotter.Start(ctx)
return errNonBatchedNotImplemented
}
// No longer need routing table handler - PrefixHandler maintains live_routes table
// Start streaming
if err := rw.streamer.Start(); err != nil {
err := rw.streamer.Start(ctx)
if err != nil {
return err
}
// Start HTTP server
if err := rw.server.Start(); err != nil {
err = rw.server.Start(ctx)
if err != nil {
return err
}
// Start ASN WHOIS fetcher for background updates
rw.asnFetcher = NewASNFetcher(rw.db, rw.logger.Logger)
rw.asnFetcher.Start(ctx)
rw.server.SetASNFetcher(rw.asnFetcher)
// Start database maintenance goroutine
rw.dbMaintainer = NewDBMaintainer(rw.db, rw.logger.Logger)
rw.dbMaintainer.Start(ctx)
// Wait for context cancellation
<-ctx.Done()
@@ -137,24 +136,60 @@ func (rw *RouteWatch) Run(ctx context.Context) error {
}
// Shutdown performs graceful shutdown of all services
func (rw *RouteWatch) Shutdown() {
func (rw *RouteWatch) Shutdown(ctx context.Context) {
rw.mu.Lock()
if rw.shutdown {
rw.mu.Unlock()
return
}
rw.shutdown = true
rw.mu.Unlock()
// Stop batched handlers first to flush remaining batches
if rw.dbHandler != nil {
rw.logger.Info("Flushing database handler")
rw.dbHandler.Stop()
}
if rw.peerHandler != nil {
rw.logger.Info("Flushing peer handler")
rw.peerHandler.Stop()
}
if rw.prefixHandler != nil {
rw.logger.Info("Flushing prefix handler")
rw.prefixHandler.Stop()
}
if rw.peeringHandler != nil {
rw.logger.Info("Flushing peering handler")
rw.peeringHandler.Stop()
}
// Stop ASN WHOIS fetcher
if rw.asnFetcher != nil {
rw.asnFetcher.Stop()
}
// Stop database maintainer
if rw.dbMaintainer != nil {
rw.dbMaintainer.Stop()
}
// Stop services
rw.streamer.Stop()
// Stop HTTP server with a timeout
// Stop HTTP server with a timeout of its own, however much of ctx is left
const serverStopTimeout = 5 * time.Second
stopCtx, cancel := context.WithTimeout(context.Background(), serverStopTimeout)
stopCtx, cancel := context.WithTimeout(
context.WithoutCancel(ctx), serverStopTimeout)
defer cancel()
if err := rw.server.Stop(stopCtx); err != nil {
err := rw.server.Stop(stopCtx)
if err != nil {
rw.logger.Error("Failed to stop HTTP server gracefully", "error", err)
}
@@ -167,84 +202,22 @@ func (rw *RouteWatch) Shutdown() {
"bits_per_sec", metrics.BitsPerSec,
"duration", time.Since(metrics.ConnectedSince),
)
// Take final snapshot before shutdown if snapshotter is available
if rw.snapshotter != nil {
rw.logger.Info("Taking final snapshot before shutdown")
if err := rw.snapshotter.Shutdown(); err != nil {
rw.logger.Error("Failed to shutdown snapshotter", "error", err)
} else {
rw.logger.Info("Final snapshot completed")
}
} else {
rw.logger.Info("No snapshotter available")
}
}
// logRoutingTableStats periodically logs routing table statistics
func (rw *RouteWatch) logRoutingTableStats(ctx context.Context) {
// Log stats periodically
ticker := time.NewTicker(routingTableStatsInterval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
stats := rw.routingTable.GetDetailedStats()
rw.logger.Info("Routing table statistics",
"ipv4_routes", stats.IPv4Routes,
"ipv6_routes", stats.IPv6Routes,
"ipv4_updates_per_sec", fmt.Sprintf("%.2f", stats.IPv4UpdatesRate),
"ipv6_updates_per_sec", fmt.Sprintf("%.2f", stats.IPv6UpdatesRate),
"total_routes", stats.TotalRoutes,
"unique_prefixes", stats.UniquePrefixes,
"unique_origins", stats.UniqueOrigins,
"unique_peers", stats.UniquePeers,
)
}
}
}
// NewLogger creates a structured logger
func NewLogger() *slog.Logger {
level := slog.LevelInfo
if debug := os.Getenv("DEBUG"); strings.Contains(debug, "routewatch") {
level = slog.LevelDebug
}
opts := &slog.HandlerOptions{
Level: level,
}
var handler slog.Handler
if os.Stdout.Name() != "/dev/stdout" || os.Getenv("TERM") == "" {
// Not a terminal, use JSON
handler = slog.NewJSONHandler(os.Stdout, opts)
} else {
// Terminal, use text
handler = slog.NewTextHandler(os.Stdout, opts)
}
return slog.New(handler)
}
// getModule provides all fx dependencies
func getModule() fx.Option {
return fx.Options(
func getModule() []fx.Option {
return []fx.Option{
fx.Provide(
NewLogger,
logger.New,
config.New,
metrics.New,
fx.Annotate(
database.New,
fx.As(new(database.Store)),
),
routingtable.New,
streamer.New,
server.New,
New,
),
)
}
}
internal/routewatch/app_integration_internal_test.go
+525
View File
@@ -0,0 +1,525 @@
package routewatch
import (
"context"
"fmt"
"strings"
"sync"
"testing"
"time"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/server"
"git.eeqj.de/sneak/routewatch/internal/streamer"
"github.com/google/uuid"
)
// mockStore is a mock implementation of database.Store for testing
type mockStore struct {
mu sync.Mutex
// Counters for tracking calls
ASNCount int
PrefixCount int
PeeringCount int
RouteCount int
WithdrawalCount int
// Track unique items
ASNs map[int]*database.ASN
Prefixes map[string]*database.Prefix
Peerings map[string]bool // key is "from_to"
Routes map[string]bool // key is "prefix_origin_peer"
// Track IP versions
IPv4Prefixes int
IPv6Prefixes int
}
// newMockStore creates a new mock store
func newMockStore() *mockStore {
return &mockStore{
ASNs: make(map[int]*database.ASN),
Prefixes: make(map[string]*database.Prefix),
Peerings: make(map[string]bool),
Routes: make(map[string]bool),
}
}
// GetOrCreateASN mock implementation
func (m *mockStore) GetOrCreateASN(
number int, timestamp time.Time,
) (*database.ASN, error) {
m.mu.Lock()
defer m.mu.Unlock()
if asn, exists := m.ASNs[number]; exists {
asn.LastSeen = timestamp
return asn, nil
}
asn := &database.ASN{
ASN: number,
FirstSeen: timestamp,
LastSeen: timestamp,
}
m.ASNs[number] = asn
m.ASNCount++
return asn, nil
}
// UpdatePrefixesBatch mock implementation
func (m *mockStore) UpdatePrefixesBatch(prefixes map[string]time.Time) error {
m.mu.Lock()
defer m.mu.Unlock()
for prefix, timestamp := range prefixes {
if p, exists := m.Prefixes[prefix]; exists {
p.LastSeen = timestamp
} else {
const (
ipVersionV4 = 4
ipVersionV6 = 6
)
ipVersion := ipVersionV4
if strings.Contains(prefix, ":") {
ipVersion = ipVersionV6
}
m.Prefixes[prefix] = &database.Prefix{
ID: uuid.New(),
Prefix: prefix,
IPVersion: ipVersion,
FirstSeen: timestamp,
LastSeen: timestamp,
}
}
}
return nil
}
// GetOrCreatePrefix mock implementation
func (m *mockStore) GetOrCreatePrefix(
prefix string, timestamp time.Time,
) (*database.Prefix, error) {
m.mu.Lock()
defer m.mu.Unlock()
if p, exists := m.Prefixes[prefix]; exists {
p.LastSeen = timestamp
return p, nil
}
const (
ipVersionV4 = 4
ipVersionV6 = 6
)
ipVersion := ipVersionV4
if strings.Contains(prefix, ":") {
ipVersion = ipVersionV6
}
p := &database.Prefix{
ID: uuid.New(),
Prefix: prefix,
IPVersion: ipVersion,
FirstSeen: timestamp,
LastSeen: timestamp,
}
m.Prefixes[prefix] = p
m.PrefixCount++
if ipVersion == ipVersionV4 {
m.IPv4Prefixes++
} else {
m.IPv6Prefixes++
}
return p, nil
}
// RecordAnnouncement mock implementation
func (m *mockStore) RecordAnnouncement(_ *database.Announcement) error {
// Not tracking announcements in detail for now
return nil
}
// RecordPeering mock implementation
func (m *mockStore) RecordPeering(asA, asB int, _ time.Time) error {
m.mu.Lock()
defer m.mu.Unlock()
// Normalize
if asA > asB {
asA, asB = asB, asA
}
key := fmt.Sprintf("%d_%d", asA, asB)
if !m.Peerings[key] {
m.Peerings[key] = true
m.PeeringCount++
}
return nil
}
// UpdatePeer mock implementation
func (m *mockStore) UpdatePeer(_ string, _ int, _ string, _ time.Time) error {
// Simple mock - just return nil
return nil
}
// Close mock implementation
func (m *mockStore) Close() error {
return nil
}
// GetStats returns statistics about the mock store
func (m *mockStore) GetStats() (database.Stats, error) {
m.mu.Lock()
defer m.mu.Unlock()
return database.Stats{
ASNs: len(m.ASNs),
Prefixes: len(m.Prefixes),
IPv4Prefixes: m.IPv4Prefixes,
IPv6Prefixes: m.IPv6Prefixes,
Peerings: m.PeeringCount,
Peers: 10, // Mock peer count
}, nil
}
// GetStatsContext returns statistics about the mock store with context support
func (m *mockStore) GetStatsContext(_ context.Context) (database.Stats, error) {
return m.GetStats()
}
// UpsertLiveRoute mock implementation
func (m *mockStore) UpsertLiveRoute(_ *database.LiveRoute) error {
// Simple mock - just return nil
return nil
}
// DeleteLiveRoute mock implementation
func (m *mockStore) DeleteLiveRoute(_ string, _ int, _ string) error {
// Simple mock - just return nil
return nil
}
// GetPrefixDistribution mock implementation
func (m *mockStore) GetPrefixDistribution() (
[]database.PrefixDistribution, []database.PrefixDistribution, error,
) {
// Return empty distributions for now
return nil, nil, nil
}
// GetPrefixDistributionContext mock implementation with context support
func (m *mockStore) GetPrefixDistributionContext(_ context.Context) (
[]database.PrefixDistribution, []database.PrefixDistribution, error,
) {
return m.GetPrefixDistribution()
}
// GetLiveRouteCounts mock implementation
func (m *mockStore) GetLiveRouteCounts() (int, int, error) {
// Return mock counts
return m.RouteCount / 2, m.RouteCount / 2, nil
}
// GetLiveRouteCountsContext mock implementation with context support
func (m *mockStore) GetLiveRouteCountsContext(_ context.Context) (int, int, error) {
return m.GetLiveRouteCounts()
}
// GetASDetails mock implementation
func (m *mockStore) GetASDetails(asn int) (*database.ASN, []database.LiveRoute, error) {
m.mu.Lock()
defer m.mu.Unlock()
// Check if ASN exists
if asnInfo, exists := m.ASNs[asn]; exists {
// Return empty prefixes for now
return asnInfo, []database.LiveRoute{}, nil
}
return nil, nil, database.ErrNoRoute
}
// GetASDetailsContext mock implementation with context support
func (m *mockStore) GetASDetailsContext(
_ context.Context, asn int,
) (*database.ASN, []database.LiveRoute, error) {
return m.GetASDetails(asn)
}
// GetPrefixDetails mock implementation
func (m *mockStore) GetPrefixDetails(
_ string,
) ([]database.LiveRoute, map[int]database.ASN, error) {
// Return empty routes for now
return []database.LiveRoute{}, map[int]database.ASN{}, nil
}
// GetPrefixDetailsContext mock implementation with context support
func (m *mockStore) GetPrefixDetailsContext(
_ context.Context, prefix string,
) ([]database.LiveRoute, map[int]database.ASN, error) {
return m.GetPrefixDetails(prefix)
}
func (m *mockStore) GetRandomPrefixesByLength(
_, _, _ int,
) ([]database.LiveRoute, error) {
// Return empty routes for now
return []database.LiveRoute{}, nil
}
// GetRandomPrefixesByLengthContext mock implementation with context support
func (m *mockStore) GetRandomPrefixesByLengthContext(
_ context.Context, maskLength, ipVersion, limit int,
) ([]database.LiveRoute, error) {
return m.GetRandomPrefixesByLength(maskLength, ipVersion, limit)
}
// GetASPeers mock implementation
func (m *mockStore) GetASPeers(_ int) ([]database.ASPeer, error) {
// Return empty peers for now
return []database.ASPeer{}, nil
}
// GetASPeersContext mock implementation with context support
func (m *mockStore) GetASPeersContext(
_ context.Context, asn int,
) ([]database.ASPeer, error) {
return m.GetASPeers(asn)
}
// GetIPInfo mock implementation
func (m *mockStore) GetIPInfo(ip string) (*database.IPInfo, error) {
return m.GetIPInfoContext(context.Background(), ip)
}
// GetIPInfoContext mock implementation with context support
func (m *mockStore) GetIPInfoContext(
_ context.Context, ip string,
) (*database.IPInfo, error) {
now := time.Now()
return &database.IPInfo{
IP: ip,
Netblock: "8.8.8.0/24",
MaskLength: 24,
IPVersion: 4,
NumPeers: 3,
ASN: 15169,
Handle: "GOOGLE",
Description: "Google LLC",
CountryCode: "US",
FirstSeen: now.Add(-24 * time.Hour),
LastSeen: now,
}, nil
}
// GetNextStaleASN mock implementation
func (m *mockStore) GetNextStaleASN(_ context.Context, _ time.Duration) (int, error) {
return 0, database.ErrNoStaleASN
}
// UpdateASNWHOIS mock implementation
func (m *mockStore) UpdateASNWHOIS(
_ context.Context, _ *database.ASNWHOISUpdate,
) error {
return nil
}
// GetWHOISStats mock implementation
func (m *mockStore) GetWHOISStats(
_ context.Context, _ time.Duration,
) (*database.WHOISStats, error) {
m.mu.Lock()
defer m.mu.Unlock()
return &database.WHOISStats{
TotalASNs: len(m.ASNs),
FreshASNs: 0,
StaleASNs: 0,
NeverFetched: len(m.ASNs),
}, nil
}
// UpsertLiveRouteBatch mock implementation
func (m *mockStore) UpsertLiveRouteBatch(routes []*database.LiveRoute) error {
m.mu.Lock()
defer m.mu.Unlock()
for _, route := range routes {
// Track prefix
if _, exists := m.Prefixes[route.Prefix]; !exists {
m.Prefixes[route.Prefix] = &database.Prefix{
ID: uuid.New(),
Prefix: route.Prefix,
IPVersion: route.IPVersion,
FirstSeen: route.LastUpdated,
LastSeen: route.LastUpdated,
}
m.PrefixCount++
if route.IPVersion == 4 {
m.IPv4Prefixes++
} else {
m.IPv6Prefixes++
}
}
m.RouteCount++
}
return nil
}
// DeleteLiveRouteBatch mock implementation
func (m *mockStore) DeleteLiveRouteBatch(_ []database.LiveRouteDeletion) error {
// Simple mock - just return nil
return nil
}
// GetOrCreateASNBatch mock implementation
func (m *mockStore) GetOrCreateASNBatch(asns map[int]time.Time) error {
m.mu.Lock()
defer m.mu.Unlock()
for number, timestamp := range asns {
if _, exists := m.ASNs[number]; !exists {
m.ASNs[number] = &database.ASN{
ASN: number,
FirstSeen: timestamp,
LastSeen: timestamp,
}
m.ASNCount++
}
}
return nil
}
// UpdatePeerBatch mock implementation
func (m *mockStore) UpdatePeerBatch(_ map[string]database.PeerUpdate) error {
// Simple mock - just return nil
return nil
}
// Vacuum mock implementation
func (m *mockStore) Vacuum(_ context.Context) error {
return nil
}
// Analyze mock implementation
func (m *mockStore) Analyze(_ context.Context) error {
return nil
}
// Checkpoint mock implementation
func (m *mockStore) Checkpoint(_ context.Context) error {
return nil
}
// Ping mock implementation
func (m *mockStore) Ping(_ context.Context) error {
return nil
}
func TestRouteWatchLiveFeed(t *testing.T) {
t.Parallel()
if testing.Short() {
t.Skip("skipping live RIPE RIS network feed test in short mode; " +
"run without -short to include it")
}
// Create mock database
mockDB := newMockStore()
defer func() {
err := mockDB.Close()
if err != nil {
t.Errorf("failed to close mock database: %v", err)
}
}()
logger := logger.New()
// Create metrics tracker
metricsTracker := metrics.New()
// Create streamer
s := streamer.New(logger, metricsTracker)
// Create test config with empty state dir (no snapshot loading)
cfg := &config.Config{
StateDir: "",
MaxRuntime: 5 * time.Second,
EnableBatchedDatabaseWrites: true,
}
// Create server
srv := server.New(mockDB, s, logger, cfg)
// Create RouteWatch with 5 second limit
deps := Dependencies{
DB: mockDB,
Streamer: s,
Server: srv,
Logger: logger,
Config: cfg,
}
rw := New(deps)
// Run with context
ctx := context.Background()
go func() {
_ = rw.Run(ctx)
}()
// Wait for the configured duration
time.Sleep(5 * time.Second)
// Force peering processing for test
if rw.peeringHandler != nil {
rw.peeringHandler.ProcessPeeringsNow()
}
// Get statistics
stats, err := mockDB.GetStats()
if err != nil {
t.Fatalf("Failed to get stats: %v", err)
}
if stats.ASNs == 0 {
t.Error("Expected to receive some ASNs from live feed")
}
t.Logf("Received %d unique ASNs in 5 seconds", stats.ASNs)
if stats.Prefixes == 0 {
t.Error("Expected to receive some prefixes from live feed")
}
t.Logf("Received %d unique prefixes (%d IPv4, %d IPv6) in 5 seconds",
stats.Prefixes, stats.IPv4Prefixes, stats.IPv6Prefixes)
if stats.Peerings == 0 {
t.Error("Expected to receive some peerings from live feed")
}
t.Logf("Recorded %d AS peering relationships in 5 seconds", stats.Peerings)
}
internal/routewatch/app_integration_test.go
-228
View File
@@ -1,228 +0,0 @@
package routewatch
import (
"context"
"strings"
"sync"
"testing"
"time"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/routingtable"
"git.eeqj.de/sneak/routewatch/internal/server"
"git.eeqj.de/sneak/routewatch/internal/streamer"
"github.com/google/uuid"
)
// mockStore is a mock implementation of database.Store for testing
type mockStore struct {
mu sync.Mutex
// Counters for tracking calls
ASNCount int
PrefixCount int
PeeringCount int
RouteCount int
WithdrawalCount int
// Track unique items
ASNs map[int]*database.ASN
Prefixes map[string]*database.Prefix
Peerings map[string]bool // key is "from_to"
Routes map[string]bool // key is "prefix_origin_peer"
// Track IP versions
IPv4Prefixes int
IPv6Prefixes int
}
// newMockStore creates a new mock store
func newMockStore() *mockStore {
return &mockStore{
ASNs: make(map[int]*database.ASN),
Prefixes: make(map[string]*database.Prefix),
Peerings: make(map[string]bool),
Routes: make(map[string]bool),
}
}
// GetOrCreateASN mock implementation
func (m *mockStore) GetOrCreateASN(number int, timestamp time.Time) (*database.ASN, error) {
m.mu.Lock()
defer m.mu.Unlock()
if asn, exists := m.ASNs[number]; exists {
asn.LastSeen = timestamp
return asn, nil
}
asn := &database.ASN{
ID: uuid.New(),
Number: number,
FirstSeen: timestamp,
LastSeen: timestamp,
}
m.ASNs[number] = asn
m.ASNCount++
return asn, nil
}
// GetOrCreatePrefix mock implementation
func (m *mockStore) GetOrCreatePrefix(prefix string, timestamp time.Time) (*database.Prefix, error) {
m.mu.Lock()
defer m.mu.Unlock()
if p, exists := m.Prefixes[prefix]; exists {
p.LastSeen = timestamp
return p, nil
}
const (
ipVersionV4 = 4
ipVersionV6 = 6
)
ipVersion := ipVersionV4
if strings.Contains(prefix, ":") {
ipVersion = ipVersionV6
}
p := &database.Prefix{
ID: uuid.New(),
Prefix: prefix,
IPVersion: ipVersion,
FirstSeen: timestamp,
LastSeen: timestamp,
}
m.Prefixes[prefix] = p
m.PrefixCount++
if ipVersion == ipVersionV4 {
m.IPv4Prefixes++
} else {
m.IPv6Prefixes++
}
return p, nil
}
// RecordAnnouncement mock implementation
func (m *mockStore) RecordAnnouncement(_ *database.Announcement) error {
// Not tracking announcements in detail for now
return nil
}
// RecordPeering mock implementation
func (m *mockStore) RecordPeering(fromASNID, toASNID string, _ time.Time) error {
m.mu.Lock()
defer m.mu.Unlock()
key := fromASNID + "_" + toASNID
if !m.Peerings[key] {
m.Peerings[key] = true
m.PeeringCount++
}
return nil
}
// UpdatePeer mock implementation
func (m *mockStore) UpdatePeer(peerIP string, peerASN int, messageType string, timestamp time.Time) error {
// Simple mock - just return nil
return nil
}
// Close mock implementation
func (m *mockStore) Close() error {
return nil
}
// GetStats returns statistics about the mock store
func (m *mockStore) GetStats() (database.Stats, error) {
m.mu.Lock()
defer m.mu.Unlock()
return database.Stats{
ASNs: len(m.ASNs),
Prefixes: len(m.Prefixes),
IPv4Prefixes: m.IPv4Prefixes,
IPv6Prefixes: m.IPv6Prefixes,
Peerings: m.PeeringCount,
}, nil
}
func TestRouteWatchLiveFeed(t *testing.T) {
// Disable snapshotter for tests
t.Setenv("ROUTEWATCH_DISABLE_SNAPSHOTTER", "1")
// Create mock database
mockDB := newMockStore()
defer mockDB.Close()
logger := NewLogger()
// Create metrics tracker
metricsTracker := metrics.New()
// Create streamer
s := streamer.New(logger, metricsTracker)
// Create test config with empty state dir (no snapshot loading)
cfg := &config.Config{
StateDir: "",
MaxRuntime: 5 * time.Second,
}
// Create routing table
rt := routingtable.New(cfg, logger)
// Create server
srv := server.New(mockDB, rt, s, logger)
// Create RouteWatch with 5 second limit
deps := Dependencies{
DB: mockDB,
RoutingTable: rt,
Streamer: s,
Server: srv,
Logger: logger,
Config: cfg,
}
rw := New(deps)
// Run with context
ctx := context.Background()
go func() {
_ = rw.Run(ctx)
}()
// Wait for the configured duration
time.Sleep(5 * time.Second)
// Get statistics
stats, err := mockDB.GetStats()
if err != nil {
t.Fatalf("Failed to get stats: %v", err)
}
if stats.ASNs == 0 {
t.Error("Expected to receive some ASNs from live feed")
}
t.Logf("Received %d unique ASNs in 5 seconds", stats.ASNs)
if stats.Prefixes == 0 {
t.Error("Expected to receive some prefixes from live feed")
}
t.Logf("Received %d unique prefixes (%d IPv4, %d IPv6) in 5 seconds", stats.Prefixes, stats.IPv4Prefixes, stats.IPv6Prefixes)
if stats.Peerings == 0 {
t.Error("Expected to receive some peerings from live feed")
}
t.Logf("Recorded %d AS peering relationships in 5 seconds", stats.Peerings)
}
internal/routewatch/app_test.go
+2 -11
View File
@@ -1,12 +1,3 @@
package routewatch
package routewatch_test
import (
"testing"
)
func TestNewLogger(t *testing.T) {
logger := NewLogger()
if logger == nil {
t.Fatal("NewLogger returned nil")
}
}
// Tests for routewatch package are in app_integration_test.go
internal/routewatch/ashandler.go
+188
View File
@@ -0,0 +1,188 @@
package routewatch
import (
"sync"
"time"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
)
const (
// asHandlerQueueSize is the queue capacity for ASN operations, about 4
// seconds of feed at peak. The streamer drops rather than blocks when a
// queue is full, so this bounds memory. Batches still flush on a timer
// (asnBatchTimeout), so a queue smaller than asnBatchSize is fine.
asHandlerQueueSize = 20000
// asnBatchSize is the number of ASN operations to batch together
asnBatchSize = 30000
// asnBatchTimeout is the maximum time to wait before flushing a batch
// DO NOT reduce this timeout - larger batches are more efficient
asnBatchTimeout = 2 * time.Second
)
// ASHandler processes Autonomous System Number (ASN) information extracted from
// BGP UPDATE messages. It uses batched database operations to efficiently store
// ASN data, collecting operations into batches that are flushed either when the
// batch reaches a size threshold or after a timeout period.
type ASHandler struct {
db database.Store
logger *logger.Logger
// Batching
mu sync.Mutex
batch []asnOp
lastFlush time.Time
stopCh chan struct{}
wg sync.WaitGroup
}
type asnOp struct {
number int
timestamp time.Time
}
// NewASHandler creates and returns a new ASHandler instance. It initializes
// the batching system and starts a background goroutine that periodically
// flushes accumulated ASN operations to the database. The caller must call
// Stop when finished to ensure all pending operations are flushed and the
// background goroutine is terminated.
func NewASHandler(db database.Store, logger *logger.Logger) *ASHandler {
h := &ASHandler{
db: db,
logger: logger,
batch: make([]asnOp, 0, asnBatchSize),
lastFlush: time.Now(),
stopCh: make(chan struct{}),
}
// Start the flush timer goroutine
h.wg.Add(1)
go h.flushLoop()
return h
}
// WantsMessage reports whether this handler should process messages of the
// given type. ASHandler only processes "UPDATE" messages, as these contain
// the AS path information needed to track autonomous systems.
func (h *ASHandler) WantsMessage(messageType string) bool {
// We only care about UPDATE messages for the database
return messageType == updateMessageType
}
// QueueCapacity returns the recommended message queue size for this handler.
// ASHandler uses a large queue capacity to accommodate high-volume BGP streams,
// as the batching mechanism allows efficient processing of accumulated messages.
func (h *ASHandler) QueueCapacity() int {
// Batching allows us to use a larger queue
return asHandlerQueueSize
}
// HandleMessage processes a RIS Live BGP message by extracting all ASNs from
// the AS path and queuing them for batch insertion into the database. The
// origin ASN (last element in the path) and all transit ASNs are recorded
// with their associated timestamps. The batch is automatically flushed when
// it reaches the configured size threshold.
func (h *ASHandler) HandleMessage(msg *ristypes.RISMessage) {
// Use the pre-parsed timestamp
timestamp := msg.ParsedTimestamp
// Get origin ASN from path (last element)
var originASN int
if len(msg.Path) > 0 {
originASN = msg.Path[len(msg.Path)-1]
}
h.mu.Lock()
defer h.mu.Unlock()
// Queue origin ASN operation
if originASN > 0 {
h.batch = append(h.batch, asnOp{
number: originASN,
timestamp: timestamp,
})
}
// Also track all ASNs in the path
for _, asn := range msg.Path {
if asn > 0 {
h.batch = append(h.batch, asnOp{
number: asn,
timestamp: timestamp,
})
}
}
// Check if we need to flush
if len(h.batch) >= asnBatchSize {
h.flushBatchLocked()
}
}
// Stop gracefully shuts down the ASHandler by signaling the background flush
// goroutine to terminate and waiting for it to complete. Any pending ASN
// operations in the current batch are flushed to the database before Stop
// returns. This method should be called during application shutdown to ensure
// no data is lost.
func (h *ASHandler) Stop() {
close(h.stopCh)
h.wg.Wait()
}
// flushLoop runs in a goroutine and periodically flushes batches
func (h *ASHandler) flushLoop() {
defer h.wg.Done()
ticker := time.NewTicker(asnBatchTimeout)
defer ticker.Stop()
for {
select {
case <-ticker.C:
h.mu.Lock()
if time.Since(h.lastFlush) >= asnBatchTimeout {
h.flushBatchLocked()
}
h.mu.Unlock()
case <-h.stopCh:
// Final flush
h.mu.Lock()
h.flushBatchLocked()
h.mu.Unlock()
return
}
}
}
// flushBatchLocked flushes the ASN batch to the database (must be called with
// mutex held)
func (h *ASHandler) flushBatchLocked() {
if len(h.batch) == 0 {
return
}
// Process ASNs first (deduped)
asnMap := make(map[int]time.Time)
for _, op := range h.batch {
if existing, ok := asnMap[op.number]; !ok || op.timestamp.After(existing) {
asnMap[op.number] = op.timestamp
}
}
// Process all ASNs in a single batch transaction
err := h.db.GetOrCreateASNBatch(asnMap)
if err != nil {
h.logger.Error("Failed to process ASN batch",
"error", err, "count", len(asnMap))
}
// Clear batch
h.batch = h.batch[:0]
h.lastFlush = time.Now()
}
internal/routewatch/asnfetcher.go
+321
View File
@@ -0,0 +1,321 @@
package routewatch
import (
"context"
"errors"
"log/slog"
"sync"
"time"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/server"
"git.eeqj.de/sneak/routewatch/internal/whois"
)
// ASN fetcher configuration constants.
const (
// baseInterval is the starting interval between fetch attempts.
baseInterval = 15 * time.Second
// minInterval is the minimum interval after successes (rate limit).
minInterval = 1 * time.Second
// maxInterval is the maximum interval after failures (backoff cap).
maxInterval = 5 * time.Minute
// backoffMultiplier is how much to multiply interval on failure.
backoffMultiplier = 2
// whoisStaleThreshold is how old WHOIS data can be before refresh.
whoisStaleThreshold = 30 * 24 * time.Hour // 30 days
// immediateQueueSize is the buffer size for immediate fetch requests.
immediateQueueSize = 100
// statsWindow is how long to keep stats for.
statsWindow = time.Hour
)
// ASNFetcher handles background WHOIS lookups for ASNs.
type ASNFetcher struct {
db database.Store
whoisClient *whois.Client
logger *slog.Logger
immediateQueue chan int
stopCh chan struct{}
wg sync.WaitGroup
// fetchMu ensures only one fetch runs at a time
fetchMu sync.Mutex
// interval tracking with mutex protection
intervalMu sync.Mutex
currentInterval time.Duration
consecutiveFails int
// hourly stats tracking
statsMu sync.Mutex
successTimes []time.Time
errorTimes []time.Time
}
// NewASNFetcher creates a new ASN fetcher.
func NewASNFetcher(db database.Store, logger *slog.Logger) *ASNFetcher {
return &ASNFetcher{
db: db,
whoisClient: whois.NewClient(),
logger: logger.With("component", "asn_fetcher"),
immediateQueue: make(chan int, immediateQueueSize),
stopCh: make(chan struct{}),
currentInterval: baseInterval,
successTimes: make([]time.Time, 0),
errorTimes: make([]time.Time, 0),
}
}
// Start begins the background ASN fetcher goroutine. It runs until Stop:
// the end of ctx does not stop it.
func (f *ASNFetcher) Start(ctx context.Context) {
f.wg.Add(1)
go f.run(context.WithoutCancel(ctx))
f.logger.Info("ASN fetcher started",
"base_interval", baseInterval,
"min_interval", minInterval,
"max_interval", maxInterval,
)
}
// Stop gracefully shuts down the fetcher.
func (f *ASNFetcher) Stop() {
close(f.stopCh)
f.wg.Wait()
f.logger.Info("ASN fetcher stopped")
}
// QueueImmediate queues an ASN for immediate WHOIS lookup.
// Non-blocking - if queue is full, the request is dropped.
func (f *ASNFetcher) QueueImmediate(asn int) {
select {
case f.immediateQueue <- asn:
f.logger.Debug("Queued immediate WHOIS lookup", "asn", asn)
default:
f.logger.Debug("Immediate queue full, dropping request", "asn", asn)
}
}
// GetStats returns statistics about fetcher activity.
func (f *ASNFetcher) GetStats() server.ASNFetcherStats {
f.statsMu.Lock()
defer f.statsMu.Unlock()
f.intervalMu.Lock()
interval := f.currentInterval
fails := f.consecutiveFails
f.intervalMu.Unlock()
// Prune old entries and count
cutoff := time.Now().Add(-statsWindow)
f.successTimes = pruneOldTimes(f.successTimes, cutoff)
f.errorTimes = pruneOldTimes(f.errorTimes, cutoff)
return server.ASNFetcherStats{
SuccessesLastHour: len(f.successTimes),
ErrorsLastHour: len(f.errorTimes),
CurrentInterval: interval,
ConsecutiveFails: fails,
}
}
// pruneOldTimes removes times older than cutoff and returns the pruned slice.
func pruneOldTimes(times []time.Time, cutoff time.Time) []time.Time {
result := make([]time.Time, 0, len(times))
for _, t := range times {
if t.After(cutoff) {
result = append(result, t)
}
}
return result
}
// getInterval returns the current fetch interval.
func (f *ASNFetcher) getInterval() time.Duration {
f.intervalMu.Lock()
defer f.intervalMu.Unlock()
return f.currentInterval
}
// recordSuccess decreases the interval on successful fetch.
func (f *ASNFetcher) recordSuccess() {
f.intervalMu.Lock()
f.consecutiveFails = 0
// Decrease interval by half, but not below minimum
newInterval := max(f.currentInterval/backoffMultiplier, minInterval)
if newInterval != f.currentInterval {
f.logger.Debug("Decreased fetch interval",
"old_interval", f.currentInterval,
"new_interval", newInterval,
)
f.currentInterval = newInterval
}
f.intervalMu.Unlock()
// Record success time for stats
f.statsMu.Lock()
f.successTimes = append(f.successTimes, time.Now())
f.statsMu.Unlock()
}
// recordFailure increases the interval on failed fetch using exponential backoff.
func (f *ASNFetcher) recordFailure() {
f.intervalMu.Lock()
f.consecutiveFails++
// Exponential backoff: multiply by 2, capped at max
newInterval := min(f.currentInterval*backoffMultiplier, maxInterval)
if newInterval != f.currentInterval {
f.logger.Debug("Increased fetch interval due to failure",
"old_interval", f.currentInterval,
"new_interval", newInterval,
"consecutive_failures", f.consecutiveFails,
)
f.currentInterval = newInterval
}
f.intervalMu.Unlock()
// Record error time for stats
f.statsMu.Lock()
f.errorTimes = append(f.errorTimes, time.Now())
f.statsMu.Unlock()
}
// run is the main background loop.
func (f *ASNFetcher) run(ctx context.Context) {
defer f.wg.Done()
timer := time.NewTimer(f.getInterval())
defer timer.Stop()
for {
select {
case <-f.stopCh:
return
case asn := <-f.immediateQueue:
// Process immediate request (respects lock)
f.tryFetch(ctx, asn)
// Reset timer after immediate fetch
timer.Reset(f.getInterval())
case <-timer.C:
// Background fetch of stale/missing ASN
f.fetchNextStale(ctx)
// Reset timer with potentially updated interval
timer.Reset(f.getInterval())
}
}
}
// tryFetch attempts to fetch and update an ASN, respecting the fetch lock.
// Returns true if fetch was successful.
func (f *ASNFetcher) tryFetch(ctx context.Context, asn int) bool {
// Try to acquire lock, skip if another fetch is running
if !f.fetchMu.TryLock() {
f.logger.Debug("Skipping fetch, another fetch in progress", "asn", asn)
return false
}
defer f.fetchMu.Unlock()
return f.fetchAndUpdate(ctx, asn)
}
// fetchNextStale finds and fetches the next ASN needing WHOIS data.
func (f *ASNFetcher) fetchNextStale(ctx context.Context) {
// Try to acquire lock, skip if another fetch is running
if !f.fetchMu.TryLock() {
f.logger.Debug("Skipping stale fetch, another fetch in progress")
return
}
defer f.fetchMu.Unlock()
lookupCtx, cancel := context.WithTimeout(ctx, time.Minute)
defer cancel()
asn, err := f.db.GetNextStaleASN(lookupCtx, whoisStaleThreshold)
if err != nil {
if !errors.Is(err, database.ErrNoStaleASN) {
f.logger.Error("Failed to get stale ASN", "error", err)
f.recordFailure()
}
// No stale ASN is not a failure, just nothing to do
return
}
f.fetchAndUpdate(ctx, asn)
}
// fetchAndUpdate performs a WHOIS lookup and updates the database.
// Returns true if successful.
func (f *ASNFetcher) fetchAndUpdate(ctx context.Context, asn int) bool {
ctx, cancel := context.WithTimeout(ctx, time.Minute)
defer cancel()
f.logger.Info("Fetching WHOIS data", "asn", asn)
info, err := f.whoisClient.LookupASN(ctx, asn)
if err != nil {
f.logger.Error("WHOIS lookup failed", "asn", asn, "error", err)
f.recordFailure()
return false
}
// Update database with WHOIS data
err = f.db.UpdateASNWHOIS(ctx, &database.ASNWHOISUpdate{
ASN: asn,
ASName: info.ASName,
OrgName: info.OrgName,
OrgID: info.OrgID,
Address: info.Address,
CountryCode: info.CountryCode,
AbuseEmail: info.AbuseEmail,
AbusePhone: info.AbusePhone,
TechEmail: info.TechEmail,
TechPhone: info.TechPhone,
RIR: info.RIR,
RIRRegDate: info.RegDate,
RIRLastMod: info.LastMod,
WHOISRaw: info.RawResponse,
})
if err != nil {
f.logger.Error("Failed to update ASN WHOIS data", "asn", asn, "error", err)
f.recordFailure()
return false
}
f.recordSuccess()
f.logger.Info("Updated ASN WHOIS data",
"asn", asn,
"org_name", info.OrgName,
"country", info.CountryCode,
"rir", info.RIR,
"next_interval", f.getInterval(),
)
return true
}
// GetStaleThreshold returns the WHOIS stale threshold duration.
func GetStaleThreshold() time.Duration {
return whoisStaleThreshold
}
internal/routewatch/cli.go
+71 -12
View File
@@ -2,31 +2,78 @@ package routewatch
import (
"context"
"log/slog"
"os"
"os/signal"
"runtime"
"strings"
"syscall"
"time"
"git.eeqj.de/sneak/routewatch/internal/logger"
"go.uber.org/fx"
)
const (
// shutdownTimeout is the maximum time allowed for graceful shutdown
shutdownTimeout = 60 * time.Second
// debugInterval is how often to log debug stats
debugInterval = 60 * time.Second
// bytesPerMB is bytes per megabyte
bytesPerMB = 1024 * 1024
// failureExitCode is the exit status when RouteWatch stops on an error
failureExitCode = 1
)
// CLIEntry is the main entry point for the CLI
// logDebugStats logs goroutine count and memory usage
func logDebugStats(logger *logger.Logger) {
// Only run if DEBUG env var contains "routewatch"
debugEnv := os.Getenv("DEBUG")
if !strings.Contains(debugEnv, "routewatch") {
return
}
ticker := time.NewTicker(debugInterval)
defer ticker.Stop()
for range ticker.C {
var m runtime.MemStats
runtime.ReadMemStats(&m)
logger.Debug("System stats",
"goroutines", runtime.NumGoroutine(),
"alloc_mb", m.Alloc/bytesPerMB,
"total_alloc_mb", m.TotalAlloc/bytesPerMB,
"sys_mb", m.Sys/bytesPerMB,
"num_gc", m.NumGC,
"heap_alloc_mb", m.HeapAlloc/bytesPerMB,
"heap_sys_mb", m.HeapSys/bytesPerMB,
"heap_idle_mb", m.HeapIdle/bytesPerMB,
"heap_inuse_mb", m.HeapInuse/bytesPerMB,
"heap_released_mb", m.HeapReleased/bytesPerMB,
"stack_inuse_mb", m.StackInuse/bytesPerMB,
)
}
}
// CLIEntry is the main entry point for the routewatch command-line interface.
// It initializes the application using the fx dependency injection framework,
// sets up signal handling for graceful shutdown, and starts the RouteWatch
// service. This function blocks until the application receives a shutdown
// signal or encounters a fatal error.
func CLIEntry() {
app := fx.New(
getModule(),
fx.Options(getModule()...),
fx.StopTimeout(shutdownTimeout), // Allow 60 seconds for graceful shutdown
fx.Invoke(func(lc fx.Lifecycle, rw *RouteWatch, logger *slog.Logger) {
fx.Invoke(func(
lc fx.Lifecycle,
rw *RouteWatch,
logger *logger.Logger,
shutdowner fx.Shutdowner,
) {
lc.Append(fx.Hook{
OnStart: func(_ context.Context) error {
go func() {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
OnStart: func(ctx context.Context) error {
// Start debug stats logging
go logDebugStats(logger)
// Handle shutdown signals
sigCh := make(chan os.Signal, 1)
@@ -35,19 +82,31 @@ func CLIEntry() {
go func() {
<-sigCh
logger.Info("Received shutdown signal")
cancel()
err := shutdowner.Shutdown()
if err != nil {
logger.Error("Failed to shutdown gracefully", "error", err)
}
}()
if err := rw.Run(ctx); err != nil {
go func() {
err := rw.Run(ctx)
if err != nil {
logger.Error("RouteWatch error", "error", err)
err = shutdowner.Shutdown(fx.ExitCode(failureExitCode))
if err != nil {
logger.Error("Failed to shutdown gracefully",
"error", err)
}
}
}()
return nil
},
OnStop: func(_ context.Context) error {
OnStop: func(ctx context.Context) error {
logger.Info("Shutting down RouteWatch")
rw.Shutdown()
rw.Shutdown(ctx)
return nil
},
internal/routewatch/dbhandler.go
-156
View File
@@ -1,156 +0,0 @@
package routewatch
import (
"log/slog"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
)
const (
// databaseHandlerQueueSize is the queue capacity for database operations
databaseHandlerQueueSize = 200
)
// DatabaseHandler handles BGP messages and stores them in the database
type DatabaseHandler struct {
db database.Store
logger *slog.Logger
}
// NewDatabaseHandler creates a new database handler
func NewDatabaseHandler(
db database.Store,
logger *slog.Logger,
) *DatabaseHandler {
return &DatabaseHandler{
db: db,
logger: logger,
}
}
// WantsMessage returns true if this handler wants to process messages of the given type
func (h *DatabaseHandler) WantsMessage(messageType string) bool {
// We only care about UPDATE messages for the database
return messageType == "UPDATE"
}
// QueueCapacity returns the desired queue capacity for this handler
func (h *DatabaseHandler) QueueCapacity() int {
// Database operations are slow, so use a smaller queue
return databaseHandlerQueueSize
}
// HandleMessage processes a RIS message and updates the database
func (h *DatabaseHandler) HandleMessage(msg *ristypes.RISMessage) {
// Use the pre-parsed timestamp
timestamp := msg.ParsedTimestamp
// Get origin ASN from path (last element)
var originASN int
if len(msg.Path) > 0 {
originASN = msg.Path[len(msg.Path)-1]
}
// Process announcements
for _, announcement := range msg.Announcements {
for _, prefix := range announcement.Prefixes {
// Get or create prefix
_, err := h.db.GetOrCreatePrefix(prefix, timestamp)
if err != nil {
h.logger.Error(
"Failed to get/create prefix",
"prefix",
prefix,
"error",
err,
)
continue
}
// Get or create origin ASN
_, err = h.db.GetOrCreateASN(originASN, timestamp)
if err != nil {
h.logger.Error(
"Failed to get/create ASN",
"asn",
originASN,
"error",
err,
)
continue
}
// TODO: Record the announcement in the announcements table
// Process AS path to update peerings
if len(msg.Path) > 1 {
for i := range len(msg.Path) - 1 {
fromASN := msg.Path[i]
toASN := msg.Path[i+1]
// Get or create both ASNs
fromAS, err := h.db.GetOrCreateASN(fromASN, timestamp)
if err != nil {
h.logger.Error(
"Failed to get/create from ASN",
"asn",
fromASN,
"error",
err,
)
continue
}
toAS, err := h.db.GetOrCreateASN(toASN, timestamp)
if err != nil {
h.logger.Error(
"Failed to get/create to ASN",
"asn",
toASN,
"error",
err,
)
continue
}
// Record the peering
err = h.db.RecordPeering(
fromAS.ID.String(),
toAS.ID.String(),
timestamp,
)
if err != nil {
h.logger.Error("Failed to record peering",
"from_asn", fromASN,
"to_asn", toASN,
"error", err,
)
}
}
}
}
}
// Process withdrawals
for _, prefix := range msg.Withdrawals {
// Get prefix
_, err := h.db.GetOrCreatePrefix(prefix, timestamp)
if err != nil {
h.logger.Error(
"Failed to get prefix for withdrawal",
"prefix",
prefix,
"error",
err,
)
continue
}
// TODO: Record the withdrawal in the announcements table as a withdrawal
}
}
internal/routewatch/dbmaintainer.go
+191
View File
@@ -0,0 +1,191 @@
package routewatch
import (
"context"
"log/slog"
"sync"
"time"
"git.eeqj.de/sneak/routewatch/internal/database"
)
// Database maintenance configuration constants.
const (
// checkpointInterval is how often to run WAL checkpoint.
// Frequent checkpoints keep the WAL small, improving read performance.
// Under heavy write load, we need aggressive checkpointing.
checkpointInterval = 5 * time.Second
// vacuumInterval is how often to run incremental vacuum.
// Since incremental vacuum only frees ~1000 pages (~4MB) per run,
// we run it frequently to keep up with deletions.
vacuumInterval = 10 * time.Minute
// analyzeInterval is how often to run ANALYZE.
analyzeInterval = 1 * time.Hour
// checkpointTimeout is the max time for WAL checkpoint.
checkpointTimeout = 10 * time.Second
// vacuumTimeout is the max time for incremental vacuum (should be quick).
vacuumTimeout = 30 * time.Second
// analyzeTimeout is the max time for ANALYZE.
analyzeTimeout = 5 * time.Minute
)
// DBMaintainer handles background database maintenance tasks.
type DBMaintainer struct {
db database.Store
logger *slog.Logger
stopCh chan struct{}
wg sync.WaitGroup
// Stats tracking
statsMu sync.Mutex
lastCheckpoint time.Time
lastVacuum time.Time
lastAnalyze time.Time
checkpointCount int
vacuumCount int
analyzeCount int
lastCheckpointError error
lastVacuumError error
lastAnalyzeError error
}
// NewDBMaintainer creates a new database maintainer.
func NewDBMaintainer(db database.Store, logger *slog.Logger) *DBMaintainer {
return &DBMaintainer{
db: db,
logger: logger.With("component", "db_maintainer"),
stopCh: make(chan struct{}),
}
}
// Start begins the background maintenance goroutine. It runs until Stop:
// the end of ctx does not stop it.
func (m *DBMaintainer) Start(ctx context.Context) {
m.wg.Add(1)
go m.run(context.WithoutCancel(ctx))
m.logger.Info("Database maintainer started",
"checkpoint_interval", checkpointInterval,
"vacuum_interval", vacuumInterval,
"analyze_interval", analyzeInterval,
)
}
// Stop gracefully shuts down the maintainer.
func (m *DBMaintainer) Stop() {
close(m.stopCh)
m.wg.Wait()
m.logger.Info("Database maintainer stopped")
}
// run is the main background loop.
func (m *DBMaintainer) run(ctx context.Context) {
defer m.wg.Done()
// Use different timers for each task
checkpointTimer := time.NewTimer(checkpointInterval)
defer checkpointTimer.Stop()
vacuumTimer := time.NewTimer(vacuumInterval)
defer vacuumTimer.Stop()
analyzeTimer := time.NewTimer(analyzeInterval)
defer analyzeTimer.Stop()
for {
select {
case <-m.stopCh:
return
case <-checkpointTimer.C:
m.runCheckpoint(ctx)
checkpointTimer.Reset(checkpointInterval)
case <-vacuumTimer.C:
m.runVacuum(ctx)
vacuumTimer.Reset(vacuumInterval)
case <-analyzeTimer.C:
m.runAnalyze(ctx)
analyzeTimer.Reset(analyzeInterval)
}
}
}
// runCheckpoint performs a WAL checkpoint to keep the WAL file small.
func (m *DBMaintainer) runCheckpoint(ctx context.Context) {
duration, err := runWithTimeout(ctx, checkpointTimeout, m.db.Checkpoint)
m.recordResult(&m.lastCheckpoint, &m.lastCheckpointError, &m.checkpointCount, err)
if err != nil {
m.logger.Error("WAL checkpoint failed", "error", err, "duration", duration)
} else {
m.logger.Debug("WAL checkpoint completed", "duration", duration)
}
}
// runVacuum performs an incremental vacuum operation on the database.
func (m *DBMaintainer) runVacuum(ctx context.Context) {
m.logger.Debug("Running incremental vacuum")
duration, err := runWithTimeout(ctx, vacuumTimeout, m.db.Vacuum)
m.recordResult(&m.lastVacuum, &m.lastVacuumError, &m.vacuumCount, err)
if err != nil {
m.logger.Error("Incremental vacuum failed", "error", err, "duration", duration)
} else {
m.logger.Debug("Incremental vacuum completed", "duration", duration)
}
}
// runAnalyze performs an ANALYZE operation on the database.
func (m *DBMaintainer) runAnalyze(ctx context.Context) {
m.logger.Info("Starting database ANALYZE")
duration, err := runWithTimeout(ctx, analyzeTimeout, m.db.Analyze)
m.recordResult(&m.lastAnalyze, &m.lastAnalyzeError, &m.analyzeCount, err)
if err != nil {
m.logger.Error("ANALYZE failed", "error", err, "duration", duration)
} else {
m.logger.Info("ANALYZE completed", "duration", duration)
}
}
// runWithTimeout runs operation with a context that ends after timeout and
// returns how long it took and its error.
func runWithTimeout(
ctx context.Context,
timeout time.Duration,
operation func(context.Context) error,
) (time.Duration, error) {
ctx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
startTime := time.Now()
err := operation(ctx)
return time.Since(startTime), err
}
// recordResult stores when a maintenance task last ran and its error, and
// counts it when it succeeded.
func (m *DBMaintainer) recordResult(
lastRun *time.Time, lastErr *error, successCount *int, err error,
) {
m.statsMu.Lock()
defer m.statsMu.Unlock()
*lastRun = time.Now()
*lastErr = err
if err == nil {
*successCount++
}
}
internal/routewatch/handler.go
+29 -14
View File
@@ -1,19 +1,36 @@
package routewatch
import (
"slices"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
"log/slog"
)
// SimpleHandler is a basic implementation of streamer.MessageHandler
// updateMessageType is the RIS message type of BGP UPDATE messages, the only
// type the database handlers process.
const updateMessageType = "UPDATE"
// SimpleHandler is a basic implementation of streamer.MessageHandler that
// filters messages by type and delegates processing to a callback function.
// It provides a simple way to handle specific RIS message types without
// implementing the full MessageHandler interface from scratch.
type SimpleHandler struct {
logger *slog.Logger
logger *logger.Logger
messageTypes []string
callback func(*ristypes.RISMessage)
}
// NewSimpleHandler creates a handler that accepts specific message types
func NewSimpleHandler(logger *slog.Logger, messageTypes []string, callback func(*ristypes.RISMessage)) *SimpleHandler {
// NewSimpleHandler creates a new SimpleHandler that accepts specific message
// types. The messageTypes parameter specifies which RIS message types this
// handler will process. If messageTypes is empty, the handler will accept all
// message types. The callback function is invoked for each message that
// passes the type filter.
func NewSimpleHandler(
logger *logger.Logger,
messageTypes []string,
callback func(*ristypes.RISMessage),
) *SimpleHandler {
return &SimpleHandler{
logger: logger,
messageTypes: messageTypes,
@@ -21,23 +38,21 @@ func NewSimpleHandler(logger *slog.Logger, messageTypes []string, callback func(
}
}
// WantsMessage returns true if this handler wants to process messages of the given type
// WantsMessage returns true if this handler wants to process messages of the
// given type. It checks whether messageType is in the handler's configured
// list of accepted types. If no specific types were configured (empty
// messageTypes slice), it returns true for all types.
func (h *SimpleHandler) WantsMessage(messageType string) bool {
// If no specific types are set, accept all messages
if len(h.messageTypes) == 0 {
return true
}
for _, t := range h.messageTypes {
if t == messageType {
return true
}
}
return false
return slices.Contains(h.messageTypes, messageType)
}
// HandleMessage processes a RIS message
// HandleMessage processes a RIS message by invoking the configured callback function.
// If no callback was provided during construction, the message is silently ignored.
func (h *SimpleHandler) HandleMessage(msg *ristypes.RISMessage) {
if h.callback != nil {
h.callback(msg)
internal/routewatch/peerhandler.go
+151 -21
View File
@@ -1,60 +1,190 @@
package routewatch
// peerhandler.go provides batched peer tracking functionality for BGP route
// monitoring. It tracks BGP peers from all incoming RIS messages and
// maintains peer state in the database.
import (
"log/slog"
"strconv"
"sync"
"time"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
)
const (
// peerHandlerQueueSize is the queue capacity for peer tracking operations
peerHandlerQueueSize = 500
// peerHandlerQueueSize is the queue capacity for peer tracking operations,
// about 4 seconds of feed at peak. The streamer drops rather than blocks
// when a queue is full, so this bounds memory. Batches still flush on a
// timer (peerBatchTimeout).
peerHandlerQueueSize = 20000
// peerBatchSize is the number of peer updates to batch together
peerBatchSize = 10000
// peerBatchTimeout is the maximum time to wait before flushing a batch
peerBatchTimeout = 2 * time.Second
)
// PeerHandler tracks BGP peers from all message types
// PeerHandler tracks BGP peers from all message types using batched operations.
// It maintains a queue of peer updates and periodically flushes them to the database
// in batches to improve performance. The handler deduplicates peer updates within
// each batch, keeping only the most recent update for each peer IP address.
type PeerHandler struct {
db database.Store
logger *slog.Logger
logger *logger.Logger
// Batching
mu sync.Mutex
peerBatch []peerUpdate
lastFlush time.Time
stopCh chan struct{}
wg sync.WaitGroup
}
// NewPeerHandler creates a new peer tracking handler
func NewPeerHandler(db database.Store, logger *slog.Logger) *PeerHandler {
return &PeerHandler{
type peerUpdate struct {
peerIP string
peerASN int
messageType string
timestamp time.Time
}
// NewPeerHandler creates a new PeerHandler with the given database store and logger.
// It initializes the peer batch buffer and starts a background goroutine that
// periodically flushes accumulated peer updates to the database. The handler
// should be stopped by calling Stop when it is no longer needed.
func NewPeerHandler(db database.Store, logger *logger.Logger) *PeerHandler {
h := &PeerHandler{
db: db,
logger: logger,
peerBatch: make([]peerUpdate, 0, peerBatchSize),
lastFlush: time.Now(),
stopCh: make(chan struct{}),
}
// Start the flush timer goroutine
h.wg.Add(1)
go h.flushLoop()
return h
}
// WantsMessage returns true for all message types since we track peers from all messages
// WantsMessage returns true for all message types since peer information
// is extracted from every RIS message regardless of type. This satisfies
// the MessageHandler interface.
func (h *PeerHandler) WantsMessage(_ string) bool {
return true
}
// QueueCapacity returns the desired queue capacity for this handler
// QueueCapacity returns the desired queue capacity for this handler.
// The PeerHandler uses a large queue capacity because batching allows
// for efficient processing of many updates at once.
func (h *PeerHandler) QueueCapacity() int {
// Peer tracking is lightweight but involves database ops, use moderate queue
// Batching allows us to use a larger queue
return peerHandlerQueueSize
}
// HandleMessage processes a message to track peer information
// HandleMessage processes a RIS message to track peer information.
// It extracts the peer IP address and ASN from the message and adds
// the update to an internal batch. When the batch reaches peerBatchSize
// or the batch timeout expires, the batch is flushed to the database.
func (h *PeerHandler) HandleMessage(msg *ristypes.RISMessage) {
// Parse peer ASN from string
peerASN := 0
if msg.PeerASN != "" {
if asn, err := strconv.Atoi(msg.PeerASN); err == nil {
asn, err := strconv.Atoi(msg.PeerASN)
if err == nil {
peerASN = asn
}
}
// Update peer in database
if err := h.db.UpdatePeer(msg.Peer, peerASN, msg.Type, msg.ParsedTimestamp); err != nil {
h.logger.Error("Failed to update peer",
"peer", msg.Peer,
"peer_asn", peerASN,
"message_type", msg.Type,
"error", err,
)
h.mu.Lock()
defer h.mu.Unlock()
// Add to batch
h.peerBatch = append(h.peerBatch, peerUpdate{
peerIP: msg.Peer,
peerASN: peerASN,
messageType: msg.Type,
timestamp: msg.ParsedTimestamp,
})
// Check if we need to flush
if len(h.peerBatch) >= peerBatchSize {
h.flushBatchLocked()
}
}
// Stop gracefully stops the handler and flushes remaining batches
func (h *PeerHandler) Stop() {
close(h.stopCh)
h.wg.Wait()
}
// flushLoop runs in a goroutine and periodically flushes batches
func (h *PeerHandler) flushLoop() {
defer h.wg.Done()
ticker := time.NewTicker(peerBatchTimeout)
defer ticker.Stop()
for {
select {
case <-ticker.C:
h.mu.Lock()
if time.Since(h.lastFlush) >= peerBatchTimeout {
h.flushBatchLocked()
}
h.mu.Unlock()
case <-h.stopCh:
// Final flush
h.mu.Lock()
h.flushBatchLocked()
h.mu.Unlock()
return
}
}
}
// flushBatchLocked flushes the peer batch to the database (must be called
// with mutex held)
func (h *PeerHandler) flushBatchLocked() {
if len(h.peerBatch) == 0 {
return
}
// Deduplicate by peer IP, keeping the latest update for each peer
peerMap := make(map[string]peerUpdate)
for _, update := range h.peerBatch {
existing, ok := peerMap[update.peerIP]
if !ok || update.timestamp.After(existing.timestamp) {
peerMap[update.peerIP] = update
}
}
// Convert to database format
dbPeerMap := make(map[string]database.PeerUpdate)
for peerIP, update := range peerMap {
dbPeerMap[peerIP] = database.PeerUpdate{
PeerIP: update.peerIP,
PeerASN: update.peerASN,
MessageType: update.messageType,
Timestamp: update.timestamp,
}
}
// Process all peers in a single batch transaction
err := h.db.UpdatePeerBatch(dbPeerMap)
if err != nil {
h.logger.Error("Failed to process peer batch",
"error", err, "count", len(dbPeerMap))
}
// Clear batch
h.peerBatch = h.peerBatch[:0]
h.lastFlush = time.Now()
}
internal/routewatch/peeringhandler.go
+235
View File
@@ -0,0 +1,235 @@
package routewatch
import (
"encoding/json"
"sync"
"time"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
)
const (
// peeringHandlerQueueSize is the buffer capacity for the peering handler's
// message queue, about 4 seconds of feed at peak. The streamer drops
// rather than blocks when a queue is full, so this bounds memory.
peeringHandlerQueueSize = 20000
// minPathLengthForPeering specifies the minimum number of ASNs required
// in a BGP AS path to extract peering relationships. A path with fewer
// than 2 ASNs cannot contain any peering information.
minPathLengthForPeering = 2
// peeringProcessInterval controls how frequently the handler processes
// accumulated AS paths and extracts peering relationships to store
// in the database.
peeringProcessInterval = 30 * time.Second
// maxTrackedPaths bounds how many distinct AS paths are held in memory
// between processing runs. Once the map is full, further new paths are
// dropped and counted until the next run empties it.
maxTrackedPaths = 500000
)
// PeeringHandler processes BGP UPDATE messages to extract and track
// AS peering relationships. It accumulates AS paths in memory and
// periodically processes them to extract unique peering pairs, which
// are then stored in the database. The handler implements the Handler
// interface for integration with the message processing pipeline.
type PeeringHandler struct {
db database.Store
logger *logger.Logger
// In-memory AS path tracking
mu sync.Mutex
asPaths map[string]time.Time // key is JSON-encoded AS path
droppedPaths int // paths dropped because the map was full
stopCh chan struct{}
}
// NewPeeringHandler creates and initializes a new PeeringHandler with the
// provided database store and logger. It starts one background goroutine that
// periodically processes accumulated AS paths into peering records. The
// handler begins processing immediately upon creation.
func NewPeeringHandler(db database.Store, logger *logger.Logger) *PeeringHandler {
h := &PeeringHandler{
db: db,
logger: logger,
asPaths: make(map[string]time.Time),
stopCh: make(chan struct{}),
}
// Start the periodic processing goroutine
go h.processLoop()
return h
}
// WantsMessage reports whether the handler should receive messages of the
// given type. PeeringHandler only processes UPDATE messages, as these contain
// the AS path information needed to extract peering relationships.
func (h *PeeringHandler) WantsMessage(messageType string) bool {
// We only care about UPDATE messages that have AS paths
return messageType == updateMessageType
}
// QueueCapacity returns the buffer size for the handler's message queue.
// This value is used by the message dispatcher to allocate the channel
// buffer when registering the handler.
func (h *PeeringHandler) QueueCapacity() int {
return peeringHandlerQueueSize
}
// HandleMessage processes a BGP UPDATE message by storing its AS path
// in memory for later batch processing. Messages with AS paths shorter
// than minPathLengthForPeering are ignored as they cannot contain valid
// peering information.
func (h *PeeringHandler) HandleMessage(msg *ristypes.RISMessage) {
// Skip if no AS path or only one AS
if len(msg.Path) < minPathLengthForPeering {
return
}
timestamp := msg.ParsedTimestamp
// Encode AS path as JSON for use as map key
pathJSON, err := json.Marshal(msg.Path)
if err != nil {
h.logger.Error("Failed to encode AS path", "error", err)
return
}
key := string(pathJSON)
h.mu.Lock()
if _, exists := h.asPaths[key]; exists {
// Already tracked: refresh its timestamp.
h.asPaths[key] = timestamp
} else if len(h.asPaths) >= maxTrackedPaths {
// Map is full; drop this new path and count it.
h.droppedPaths++
} else {
h.asPaths[key] = timestamp
}
h.mu.Unlock()
}
// ProcessPeeringsNow triggers immediate processing of all accumulated AS
// paths into peering records. This bypasses the normal periodic processing
// schedule and is primarily intended for testing purposes.
func (h *PeeringHandler) ProcessPeeringsNow() {
h.processPeerings()
}
// Stop gracefully shuts down the handler by signaling the background
// goroutine to stop and performing a final synchronous processing of
// any remaining AS paths. This ensures no peering data is lost during
// shutdown.
func (h *PeeringHandler) Stop() {
close(h.stopCh)
// Process any remaining peerings synchronously
h.processPeerings()
}
// processLoop runs periodically to process AS paths into peerings
func (h *PeeringHandler) processLoop() {
ticker := time.NewTicker(peeringProcessInterval)
defer ticker.Stop()
for {
select {
case <-ticker.C:
h.processPeerings()
case <-h.stopCh:
// Final processing
h.processPeerings()
return
}
}
}
// processPeerings extracts peerings from AS paths and writes to database
func (h *PeeringHandler) processPeerings() {
// Take the accumulated paths and replace the map with a fresh empty one
// under the lock. Each path is processed exactly once and the memory is
// released, so the map never grows past a single interval's traffic.
h.mu.Lock()
paths := h.asPaths
h.asPaths = make(map[string]time.Time)
dropped := h.droppedPaths
h.mu.Unlock()
if len(paths) == 0 {
return
}
// Extract unique peerings from AS paths
type peeringKey struct {
low, high int
}
peerings := make(map[peeringKey]time.Time)
for pathJSON, timestamp := range paths {
var path []int
err := json.Unmarshal([]byte(pathJSON), &path)
if err != nil {
h.logger.Error("Failed to decode AS path", "error", err)
continue
}
// Extract peerings from path
for i := range len(path) - 1 {
asn1 := path[i]
asn2 := path[i+1]
// Skip invalid ASNs
if asn1 <= 0 || asn2 <= 0 || asn1 == asn2 {
continue
}
// Normalize: lower AS number first
low, high := asn1, asn2
if low > high {
low, high = high, low
}
key := peeringKey{low: low, high: high}
// Update timestamp if this is newer
if existing, ok := peerings[key]; !ok || timestamp.After(existing) {
peerings[key] = timestamp
}
}
}
// Record peerings in database
start := time.Now()
successCount := 0
for key, ts := range peerings {
err := h.db.RecordPeering(key.low, key.high, ts)
if err != nil {
h.logger.Error("Failed to record peering",
"as_a", key.low,
"as_b", key.high,
"error", err,
)
} else {
successCount++
}
}
h.logger.Info("Processed AS peerings",
"paths", len(paths),
"unique_peerings", len(peerings),
"success", successCount,
"dropped_paths", dropped,
"duration", time.Since(start),
)
}
internal/routewatch/peeringhandler_internal_test.go
+175
View File
@@ -0,0 +1,175 @@
package routewatch
import (
"encoding/json"
"strconv"
"sync"
"testing"
"time"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
)
const (
testASNA = 64500
testASNB = 64501
testASNC = 64502
)
// recordingStore wraps mockStore to count every RecordPeering call, so a
// test can tell how many times a peering was written across separate runs.
type recordingStore struct {
*mockStore
mu sync.Mutex
calls int
}
func (r *recordingStore) RecordPeering(asA, asB int, ts time.Time) error {
r.mu.Lock()
r.calls++
r.mu.Unlock()
return r.mockStore.RecordPeering(asA, asB, ts)
}
func (r *recordingStore) callCount() int {
r.mu.Lock()
defer r.mu.Unlock()
return r.calls
}
// newTestHandler builds a PeeringHandler without starting the periodic
// processing goroutine, so tests drive processing explicitly.
func newTestHandler(db database.Store) *PeeringHandler {
return &PeeringHandler{
db: db,
logger: logger.New(),
asPaths: make(map[string]time.Time),
stopCh: make(chan struct{}),
}
}
// snapshot returns how many paths the handler tracks and how many it dropped.
func snapshot(h *PeeringHandler) (int, int) {
h.mu.Lock()
defer h.mu.Unlock()
return len(h.asPaths), h.droppedPaths
}
func pathKey(t *testing.T, asns ...int) string {
t.Helper()
b, err := json.Marshal(ristypes.ASPath(asns))
if err != nil {
t.Fatalf("failed to marshal path: %v", err)
}
return string(b)
}
func handle(h *PeeringHandler, ts time.Time, asns ...int) {
h.HandleMessage(&ristypes.RISMessage{
Path: ristypes.ASPath(asns),
ParsedTimestamp: ts,
})
}
// TestPeeringHandlerProcessesEachRunAndEmpties verifies that a processing run
// empties the path map (the swap) and that a path seen again after a run is
// recorded in the next run too.
func TestPeeringHandlerProcessesEachRunAndEmpties(t *testing.T) {
t.Parallel()
store := &recordingStore{mockStore: newMockStore()}
h := newTestHandler(store)
now := time.Now().UTC()
// Run 1: one path, one peering recorded, map emptied afterwards.
handle(h, now, testASNA, testASNB)
h.ProcessPeeringsNow()
if tracked, _ := snapshot(h); tracked != 0 {
t.Fatalf("map not empty after first run: %d paths remain", tracked)
}
if got := store.callCount(); got != 1 {
t.Fatalf("want 1 RecordPeering call after first run, got %d", got)
}
// Run 2: the same path again is recorded again (RecordPeering upserts).
handle(h, now.Add(time.Second), testASNA, testASNB)
h.ProcessPeeringsNow()
if tracked, _ := snapshot(h); tracked != 0 {
t.Fatalf("map not empty after second run: %d paths remain", tracked)
}
if got := store.callCount(); got != 2 {
t.Fatalf("want 2 RecordPeering calls after second run, got %d", got)
}
}
// TestPeeringHandlerCapDropsAndCounts verifies that a full map drops new paths
// and counts them, while a path already tracked is refreshed rather than
// dropped.
func TestPeeringHandlerCapDropsAndCounts(t *testing.T) {
t.Parallel()
store := &recordingStore{mockStore: newMockStore()}
h := newTestHandler(store)
now := time.Now().UTC()
// Fill the map to exactly maxTrackedPaths, including one real path key so
// the "already tracked" branch can be exercised. The filler keys are never
// processed in this test, so their contents do not matter.
existing := pathKey(t, testASNA, testASNB)
h.mu.Lock()
h.asPaths[existing] = now
for i := 0; len(h.asPaths) < maxTrackedPaths; i++ {
h.asPaths[strconv.Itoa(i)] = now
}
h.mu.Unlock()
// A new path is dropped and counted because the map is full.
handle(h, now.Add(time.Second), testASNA, testASNC)
tracked, dropped := snapshot(h)
if tracked != maxTrackedPaths {
t.Fatalf("want map size %d after drop, got %d", maxTrackedPaths, tracked)
}
if dropped != 1 {
t.Fatalf("want dropped count 1, got %d", dropped)
}
// A path already tracked is refreshed, not dropped.
refreshed := now.Add(2 * time.Second)
handle(h, refreshed, testASNA, testASNB)
tracked, dropped = snapshot(h)
if tracked != maxTrackedPaths {
t.Fatalf("want map size %d after refresh, got %d", maxTrackedPaths, tracked)
}
if dropped != 1 {
t.Fatalf("want dropped count still 1 after refresh, got %d", dropped)
}
h.mu.Lock()
gotTS := h.asPaths[existing]
h.mu.Unlock()
if !gotTS.Equal(refreshed) {
t.Fatalf("existing path timestamp not refreshed: want %v, got %v",
refreshed, gotTS)
}
}
internal/routewatch/prefixhandler.go
+368
View File
@@ -0,0 +1,368 @@
package routewatch
import (
"net"
"net/netip"
"strings"
"sync"
"time"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
"github.com/google/uuid"
)
const (
// prefixHandlerQueueSize is the queue capacity for prefix tracking
// operations, about 4 seconds of feed at peak. The streamer drops rather
// than blocks when a queue is full, so this bounds memory. Batches still
// flush on a timer (prefixBatchTimeout).
prefixHandlerQueueSize = 20000
// prefixBatchSize is the number of prefix updates to batch together
prefixBatchSize = 25000
// prefixBatchTimeout is the maximum time to wait before flushing a batch
// DO NOT reduce this timeout - larger batches are more efficient
prefixBatchTimeout = 1 * time.Second
// IP version constants
ipv4Version = 4
ipv6Version = 6
)
// PrefixHandler tracks BGP prefixes and maintains a live routing table in the
// database. Routes are added on announcement and deleted on withdrawal.
type PrefixHandler struct {
db database.Store
logger *logger.Logger
metrics *metrics.Tracker
// Batching
mu sync.Mutex
batch []prefixUpdate
lastFlush time.Time
stopCh chan struct{}
wg sync.WaitGroup
}
type prefixUpdate struct {
prefix string
originASN int
peer string
messageType string
timestamp time.Time
path []int
}
// NewPrefixHandler creates a new batched prefix tracking handler
func NewPrefixHandler(db database.Store, logger *logger.Logger) *PrefixHandler {
h := &PrefixHandler{
db: db,
logger: logger,
batch: make([]prefixUpdate, 0, prefixBatchSize),
lastFlush: time.Now(),
stopCh: make(chan struct{}),
}
// Start the flush timer goroutine
h.wg.Add(1)
go h.flushLoop()
return h
}
// SetMetricsTracker sets the metrics tracker for recording route updates
func (h *PrefixHandler) SetMetricsTracker(metrics *metrics.Tracker) {
h.metrics = metrics
}
// WantsMessage returns true if this handler wants to process messages of the
// given type
func (h *PrefixHandler) WantsMessage(messageType string) bool {
// We only care about UPDATE messages for the routing table
return messageType == updateMessageType
}
// QueueCapacity returns the desired queue capacity for this handler
func (h *PrefixHandler) QueueCapacity() int {
// Batching allows us to use a larger queue
return prefixHandlerQueueSize
}
// HandleMessage processes a message to track prefix information
func (h *PrefixHandler) HandleMessage(msg *ristypes.RISMessage) {
// Use the pre-parsed timestamp
timestamp := msg.ParsedTimestamp
// Get origin ASN from path (last element)
var originASN int
if len(msg.Path) > 0 {
originASN = msg.Path[len(msg.Path)-1]
}
h.mu.Lock()
defer h.mu.Unlock()
// Withdrawals are queued before announcements, because the batch keeps
// each peer's last update for a prefix: a message that lists a prefix as
// both withdrawn and announced keeps the route (RFC 4271 section 4.3).
for _, prefix := range msg.Withdrawals {
h.batch = append(h.batch, prefixUpdate{
prefix: canonicalPrefix(prefix),
originASN: originASN, // Use the originASN from path if available
peer: msg.Peer,
messageType: "withdrawal",
timestamp: timestamp,
path: msg.Path,
})
// Record withdrawal in metrics
if h.metrics != nil {
h.metrics.RecordWithdrawal()
}
}
for _, announcement := range msg.Announcements {
for _, prefix := range announcement.Prefixes {
h.batch = append(h.batch, prefixUpdate{
prefix: canonicalPrefix(prefix),
originASN: originASN,
peer: msg.Peer,
messageType: "announcement",
timestamp: timestamp,
path: msg.Path,
})
// Record announcement in metrics
if h.metrics != nil {
h.metrics.RecordAnnouncement()
}
}
}
// Check if we need to flush
if len(h.batch) >= prefixBatchSize {
h.flushBatchLocked()
}
}
// Stop gracefully stops the handler and flushes remaining batches
func (h *PrefixHandler) Stop() {
close(h.stopCh)
h.wg.Wait()
}
// flushLoop runs in a goroutine and periodically flushes batches
func (h *PrefixHandler) flushLoop() {
defer h.wg.Done()
ticker := time.NewTicker(prefixBatchTimeout)
defer ticker.Stop()
for {
select {
case <-ticker.C:
h.mu.Lock()
if time.Since(h.lastFlush) >= prefixBatchTimeout {
h.flushBatchLocked()
}
h.mu.Unlock()
case <-h.stopCh:
// Final flush
h.mu.Lock()
h.flushBatchLocked()
h.mu.Unlock()
return
}
}
}
// flushBatchLocked flushes the prefix batch to the database (must be called
// with mutex held)
func (h *PrefixHandler) flushBatchLocked() {
if len(h.batch) == 0 {
return
}
startTime := time.Now()
batchSize := len(h.batch)
// For each prefix and peer, keep only the last update the feed delivered.
// Updates from different peers for the same prefix are all kept. Arrival
// order decides, not the timestamp: timestamps are whole seconds, so
// updates within one second carry the same one.
type prefixAndPeer struct{ prefix, peer string }
lastUpdates := make(map[prefixAndPeer]prefixUpdate)
for _, update := range h.batch {
lastUpdates[prefixAndPeer{update.prefix, update.peer}] = update
}
// Collect routes to upsert and delete
var (
routesToUpsert []*database.LiveRoute
routesToDelete []database.LiveRouteDeletion
)
// Collect unique prefixes to update
prefixesToUpdate := make(map[string]time.Time)
for _, update := range lastUpdates {
// Track prefix for both announcements and withdrawals
seen, exists := prefixesToUpdate[update.prefix]
if !exists || update.timestamp.After(seen) {
prefixesToUpdate[update.prefix] = update.timestamp
}
if update.messageType == "announcement" && update.originASN > 0 {
// Create live route for batch upsert
route := h.createLiveRoute(update)
if route != nil {
routesToUpsert = append(routesToUpsert, route)
}
} else if update.messageType == "withdrawal" {
// Parse CIDR to get IP version
_, ipVersion, err := parseCIDR(update.prefix)
if err != nil {
h.logger.Error("Failed to parse CIDR for withdrawal",
"prefix", update.prefix, "error", err)
continue
}
// Create deletion record for batch delete
routesToDelete = append(routesToDelete, database.LiveRouteDeletion{
Prefix: update.prefix,
OriginASN: update.originASN,
PeerIP: update.peer,
IPVersion: ipVersion,
})
}
}
successCount := h.writeBatch(routesToUpsert, routesToDelete, prefixesToUpdate)
elapsed := time.Since(startTime)
h.logger.Debug("Flushed prefix batch",
"batch_size", batchSize,
"unique_prefixes", len(prefixesToUpdate),
"success", successCount,
"duration_ms", elapsed.Milliseconds(),
)
// Clear batch
h.batch = h.batch[:0]
h.lastFlush = time.Now()
}
// writeBatch writes the collected route upserts, route deletions and prefix
// updates to the database and returns how many routes it wrote.
func (h *PrefixHandler) writeBatch(
routesToUpsert []*database.LiveRoute,
routesToDelete []database.LiveRouteDeletion,
prefixesToUpdate map[string]time.Time,
) int {
successCount := 0
if len(routesToUpsert) > 0 {
err := h.db.UpsertLiveRouteBatch(routesToUpsert)
if err != nil {
h.logger.Error("Failed to upsert route batch",
"error", err, "count", len(routesToUpsert))
} else {
successCount += len(routesToUpsert)
}
}
if len(routesToDelete) > 0 {
err := h.db.DeleteLiveRouteBatch(routesToDelete)
if err != nil {
h.logger.Error("Failed to delete route batch",
"error", err, "count", len(routesToDelete))
} else {
successCount += len(routesToDelete)
}
}
// Update prefix tables
if len(prefixesToUpdate) > 0 {
err := h.db.UpdatePrefixesBatch(prefixesToUpdate)
if err != nil {
h.logger.Error("Failed to update prefix batch",
"error", err, "count", len(prefixesToUpdate))
}
}
return successCount
}
// canonicalPrefix returns prefix in the text form net/netip prints, the form
// the IP lookup builds when it looks a prefix up. The feed sends IPv6
// announcements compressed ("2001:db8::/32") but IPv6 withdrawals uncompressed
// ("2001:db8:0:0:0:0:0:0/32"); stored as received, a withdrawal would not match
// the route its announcement stored. A prefix that does not parse is returned
// unchanged, and the batch flush reports it.
func canonicalPrefix(prefix string) string {
p, err := netip.ParsePrefix(prefix)
if err != nil {
return prefix
}
return p.Masked().String()
}
// parseCIDR returns the mask length and the IP version of a prefix string
func parseCIDR(prefix string) (int, int, error) {
_, ipNet, err := net.ParseCIDR(prefix)
if err != nil {
return 0, 0, err
}
ones, _ := ipNet.Mask.Size()
if strings.Contains(prefix, ":") {
return ones, ipv6Version, nil
}
return ones, ipv4Version, nil
}
// createLiveRoute creates a LiveRoute from a prefix update
func (h *PrefixHandler) createLiveRoute(update prefixUpdate) *database.LiveRoute {
// Parse CIDR to get mask length
maskLength, ipVersion, err := parseCIDR(update.prefix)
if err != nil {
h.logger.Error("Failed to parse CIDR",
"prefix", update.prefix,
"error", err,
)
return nil
}
// Track route update metrics
if h.metrics != nil {
if ipVersion == ipv4Version {
h.metrics.RecordIPv4Update()
} else {
h.metrics.RecordIPv6Update()
}
}
// Create live route record
liveRoute := &database.LiveRoute{
ID: uuid.New(),
Prefix: update.prefix,
MaskLength: maskLength,
IPVersion: ipVersion,
OriginASN: update.originASN,
PeerIP: update.peer,
ASPath: update.path,
NextHop: update.peer, // Using peer as next hop
LastUpdated: update.timestamp,
}
return liveRoute
}
internal/routewatch/prefixhandler_internal_test.go
+268
View File
@@ -0,0 +1,268 @@
package routewatch
import (
"context"
"errors"
"slices"
"testing"
"time"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
)
const (
testPeerIP = "2001:db8:ffff::1"
testV4Prefix = "192.0.2.0/24"
)
// TestPrefixHandlerStoresPrefixesTheIPLookupFinds runs announcements and
// withdrawals through the prefix handler into a real database and looks the
// addresses up. The prefixes are written the way the feed sends them: IPv6
// announcements compressed, IPv6 withdrawals uncompressed. The withdrawal must
// remove the route the announcement stored, and the IP lookup must find the
// stored prefix.
func TestPrefixHandlerStoresPrefixesTheIPLookupFinds(t *testing.T) {
t.Parallel()
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
// Built without NewPrefixHandler's flush timer, so the test flushes itself.
h := &PrefixHandler{db: db, logger: logger.New()}
flush := func() {
h.mu.Lock()
defer h.mu.Unlock()
h.flushBatchLocked()
}
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
h.HandleMessage(&ristypes.RISMessage{
Peer: testPeerIP,
Path: ristypes.ASPath{testASNA, testASNB},
ParsedTimestamp: ts,
Announcements: []ristypes.RISAnnouncement{{
NextHop: testPeerIP,
Prefixes: []string{"2001:db8:1::/48", testV4Prefix},
}},
})
flush()
for ip, want := range map[string]string{
"2001:db8:1::1": "2001:db8:1::/48",
"192.0.2.1": testV4Prefix,
} {
info, err := db.GetIPInfo(ip)
if err != nil {
t.Fatalf("GetIPInfo(%s) after announcement: %v", ip, err)
}
if info.Netblock != want || info.ASN != testASNB {
t.Errorf("GetIPInfo(%s) = %s AS%d, want %s AS%d",
ip, info.Netblock, info.ASN, want, testASNB)
}
}
h.HandleMessage(&ristypes.RISMessage{
Peer: testPeerIP,
ParsedTimestamp: ts.Add(time.Minute),
Withdrawals: []string{"2001:db8:1:0:0:0:0:0/48", testV4Prefix},
})
flush()
for _, ip := range []string{"2001:db8:1::1", "192.0.2.1"} {
info, err := db.GetIPInfo(ip)
if !errors.Is(err, database.ErrNoRoute) {
t.Errorf("GetIPInfo(%s) after withdrawal = %+v, %v; want ErrNoRoute",
ip, info, err)
}
}
}
// TestPrefixHandlerAppliesEachPeersLastUpdateInABatch runs updates for one
// prefix through the prefix handler into a real database. All updates carry the
// same timestamp, as updates within one second of the feed do. Within a batch,
// the last update each peer sent for the prefix decides whether that peer has a
// live route for it, and updates from different peers are all applied. The
// in-memory route count and prefix distribution must match the route table.
func TestPrefixHandlerAppliesEachPeersLastUpdateInABatch(t *testing.T) {
t.Parallel()
const (
prefix = "198.51.100.0/24"
prefixMaskLength = 24
otherPeerIP = "2001:db8:ffff::2"
)
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
announce := func(peer string) *ristypes.RISMessage {
return &ristypes.RISMessage{
Peer: peer,
Path: ristypes.ASPath{testASNA, testASNB},
ParsedTimestamp: ts,
Announcements: []ristypes.RISAnnouncement{
{NextHop: peer, Prefixes: []string{prefix}},
},
}
}
withdraw := func(peer string) *ristypes.RISMessage {
return &ristypes.RISMessage{
Peer: peer, ParsedTimestamp: ts, Withdrawals: []string{prefix},
}
}
announceAndWithdraw := func(peer string) *ristypes.RISMessage {
msg := announce(peer)
msg.Withdrawals = []string{prefix}
return msg
}
tests := []struct {
name string
// Each inner list is handled and then flushed as one batch.
batches [][]*ristypes.RISMessage
// The peers left with a live route for the prefix, by peer IP.
wantPeers []string
}{
{
name: "two peers announce the prefix",
batches: [][]*ristypes.RISMessage{{announce(testPeerIP), announce(otherPeerIP)}},
wantPeers: []string{testPeerIP, otherPeerIP},
},
{
name: "two peers withdraw the prefix",
batches: [][]*ristypes.RISMessage{
{announce(testPeerIP)},
{announce(otherPeerIP)},
{withdraw(testPeerIP), withdraw(otherPeerIP)},
},
},
{
name: "a peer announces and then withdraws the prefix",
batches: [][]*ristypes.RISMessage{{announce(testPeerIP), withdraw(testPeerIP)}},
},
{
name: "a peer withdraws and then re-announces the prefix",
batches: [][]*ristypes.RISMessage{
{announce(testPeerIP)},
{withdraw(testPeerIP), announce(testPeerIP)},
},
wantPeers: []string{testPeerIP},
},
{
// RFC 4271 section 4.3: the announcement counts, not the withdrawal.
name: "one message announces and withdraws the prefix",
batches: [][]*ristypes.RISMessage{
{announce(testPeerIP)},
{announceAndWithdraw(testPeerIP)},
},
wantPeers: []string{testPeerIP},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
db := handleBatches(t, tt.batches)
checkPrefixRoutes(t, db, prefix, prefixMaskLength, tt.wantPeers)
})
}
}
// handleBatches runs each batch of messages through a prefix handler on a new
// database, flushing after each batch, and returns the database. The database
// is closed when the test ends.
func handleBatches(
t *testing.T, batches [][]*ristypes.RISMessage,
) *database.Database {
t.Helper()
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
// Built without NewPrefixHandler's flush timer, so the test flushes itself.
h := &PrefixHandler{db: db, logger: logger.New()}
for _, batch := range batches {
for _, msg := range batch {
h.HandleMessage(msg)
}
h.mu.Lock()
h.flushBatchLocked()
h.mu.Unlock()
}
return db
}
// checkPrefixRoutes checks that exactly wantPeers have a live route for the
// IPv4 prefix, and that the in-memory route count and prefix distribution
// match the route table.
func checkPrefixRoutes(
t *testing.T, db *database.Database, prefix string, maskLength int,
wantPeers []string,
) {
t.Helper()
ctx := context.Background()
routes, _, err := db.GetPrefixDetailsContext(ctx, prefix)
if err != nil && !errors.Is(err, database.ErrNoRoute) {
t.Fatalf("GetPrefixDetailsContext: %v", err)
}
gotPeers := make([]string, 0, len(routes))
for _, route := range routes {
gotPeers = append(gotPeers, route.PeerIP)
}
if !slices.Equal(gotPeers, wantPeers) {
t.Errorf("peers with a live route for %s = %v, want %v",
prefix, gotPeers, wantPeers)
}
stats, err := db.GetStatsContext(ctx)
if err != nil {
t.Fatalf("GetStatsContext: %v", err)
}
tableRoutesV4, _, err := db.GetLiveRouteCountsContext(ctx)
if err != nil {
t.Fatalf("GetLiveRouteCountsContext: %v", err)
}
if stats.IPv4Routes != len(wantPeers) || tableRoutesV4 != len(wantPeers) {
t.Errorf("IPv4 live routes: %d in memory, %d in the table, want %d",
stats.IPv4Routes, tableRoutesV4, len(wantPeers))
}
var wantDistribution []database.PrefixDistribution
if len(wantPeers) > 0 {
wantDistribution = []database.PrefixDistribution{
{MaskLength: maskLength, Count: 1},
}
}
queryDistributionV4, _, err := db.GetPrefixDistributionContext(ctx)
if err != nil {
t.Fatalf("GetPrefixDistributionContext: %v", err)
}
if !slices.Equal(stats.IPv4PrefixDistribution, wantDistribution) ||
!slices.Equal(queryDistributionV4, wantDistribution) {
t.Errorf("IPv4 prefix distribution: %v in memory, %v from the table, want %v",
stats.IPv4PrefixDistribution, queryDistributionV4, wantDistribution)
}
}
internal/routewatch/routingtablehandler.go
-131
View File
@@ -1,131 +0,0 @@
package routewatch
import (
"log/slog"
"strconv"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
"git.eeqj.de/sneak/routewatch/internal/routingtable"
"github.com/google/uuid"
)
const (
// routingTableHandlerQueueSize is the queue capacity for in-memory routing table operations
routingTableHandlerQueueSize = 10000
)
// RoutingTableHandler handles BGP messages and updates the in-memory routing table
type RoutingTableHandler struct {
rt *routingtable.RoutingTable
logger *slog.Logger
}
// NewRoutingTableHandler creates a new routing table handler
func NewRoutingTableHandler(rt *routingtable.RoutingTable, logger *slog.Logger) *RoutingTableHandler {
return &RoutingTableHandler{
rt: rt,
logger: logger,
}
}
// WantsMessage returns true if this handler wants to process messages of the given type
func (h *RoutingTableHandler) WantsMessage(messageType string) bool {
// We only care about UPDATE messages for the routing table
return messageType == "UPDATE"
}
// QueueCapacity returns the desired queue capacity for this handler
func (h *RoutingTableHandler) QueueCapacity() int {
// In-memory operations are very fast, so use a large queue
return routingTableHandlerQueueSize
}
// HandleMessage processes a RIS message and updates the routing table
func (h *RoutingTableHandler) HandleMessage(msg *ristypes.RISMessage) {
// Use the pre-parsed timestamp
timestamp := msg.ParsedTimestamp
// Parse peer ASN
peerASN, err := strconv.Atoi(msg.PeerASN)
if err != nil {
h.logger.Error("Failed to parse peer ASN", "peer_asn", msg.PeerASN, "error", err)
return
}
// Get origin ASN from path (last element)
var originASN int
if len(msg.Path) > 0 {
originASN = msg.Path[len(msg.Path)-1]
}
// Process announcements
for _, announcement := range msg.Announcements {
for _, prefix := range announcement.Prefixes {
// Generate deterministic UUIDs based on the prefix and origin ASN
// This ensures consistency across restarts
prefixID := uuid.NewSHA1(uuid.NameSpaceURL, []byte(prefix))
originASNID := uuid.NewSHA1(uuid.NameSpaceOID, []byte(strconv.Itoa(originASN)))
// Create route for the routing table
route := &routingtable.Route{
PrefixID: prefixID,
Prefix: prefix,
OriginASNID: originASNID,
OriginASN: originASN,
PeerASN: peerASN,
ASPath: msg.Path,
NextHop: announcement.NextHop,
AnnouncedAt: timestamp,
}
// Add route to routing table
h.rt.AddRoute(route)
}
}
// Process withdrawals
for _, prefix := range msg.Withdrawals {
// Generate deterministic UUID for the prefix
prefixID := uuid.NewSHA1(uuid.NameSpaceURL, []byte(prefix))
// Withdraw all routes for this prefix from this peer
h.rt.WithdrawRoutesByPrefixAndPeer(prefixID, peerASN)
}
}
// GetRoutingTableStats returns statistics about the routing table
func (h *RoutingTableHandler) GetRoutingTableStats() map[string]int {
return h.rt.Stats()
}
// GetActiveRouteCount returns the number of active routes
func (h *RoutingTableHandler) GetActiveRouteCount() int {
return h.rt.Size()
}
// GetRoutesByPrefix returns all routes for a specific prefix
func (h *RoutingTableHandler) GetRoutesByPrefix(prefixID uuid.UUID) []*routingtable.Route {
return h.rt.GetRoutesByPrefix(prefixID)
}
// GetRoutesByOriginASN returns all routes originated by a specific ASN
func (h *RoutingTableHandler) GetRoutesByOriginASN(originASNID uuid.UUID) []*routingtable.Route {
return h.rt.GetRoutesByOriginASN(originASNID)
}
// GetRoutesByPeerASN returns all routes received from a specific peer ASN
func (h *RoutingTableHandler) GetRoutesByPeerASN(peerASN int) []*routingtable.Route {
return h.rt.GetRoutesByPeerASN(peerASN)
}
// GetAllRoutes returns all active routes
func (h *RoutingTableHandler) GetAllRoutes() []*routingtable.Route {
return h.rt.GetAllRoutes()
}
// ClearRoutingTable clears all routes from the routing table
func (h *RoutingTableHandler) ClearRoutingTable() {
h.rt.Clear()
h.logger.Info("Cleared routing table")
}
internal/routingtable/routingtable.go
-524
View File
@@ -1,524 +0,0 @@
// Package routingtable provides a thread-safe in-memory representation of the DFZ routing table.
package routingtable
import (
"compress/gzip"
"encoding/json"
"fmt"
"log/slog"
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"time"
"git.eeqj.de/sneak/routewatch/internal/config"
"github.com/google/uuid"
)
const (
// routeStalenessThreshold is how old a route can be before we consider it stale
// Using 30 minutes as a conservative value for snapshot loading
routeStalenessThreshold = 30 * time.Minute
// snapshotFilename is the name of the snapshot file
snapshotFilename = "routingtable.json.gz"
)
// Route represents a single route entry in the routing table
type Route struct {
PrefixID uuid.UUID `json:"prefix_id"`
Prefix string `json:"prefix"` // The actual prefix string (e.g., "10.0.0.0/8")
OriginASNID uuid.UUID `json:"origin_asn_id"`
OriginASN int `json:"origin_asn"` // The actual ASN number
PeerASN int `json:"peer_asn"`
ASPath []int `json:"as_path"` // Full AS path
NextHop string `json:"next_hop"`
AnnouncedAt time.Time `json:"announced_at"`
AddedAt time.Time `json:"added_at"` // When we added this route to our table
}
// RouteKey uniquely identifies a route in the table
type RouteKey struct {
PrefixID uuid.UUID
OriginASNID uuid.UUID
PeerASN int
}
// RoutingTable is a thread-safe in-memory routing table
type RoutingTable struct {
mu sync.RWMutex
routes map[RouteKey]*Route
// Secondary indexes for efficient lookups
byPrefix map[uuid.UUID]map[RouteKey]*Route // Routes indexed by prefix ID
byOriginASN map[uuid.UUID]map[RouteKey]*Route // Routes indexed by origin ASN ID
byPeerASN map[int]map[RouteKey]*Route // Routes indexed by peer ASN
// Metrics tracking
ipv4Routes int
ipv6Routes int
ipv4Updates uint64 // Updates counter for rate calculation
ipv6Updates uint64 // Updates counter for rate calculation
lastMetricsReset time.Time
// Configuration
snapshotDir string
}
// New creates a new routing table, loading from snapshot if available
func New(cfg *config.Config, logger *slog.Logger) *RoutingTable {
rt := &RoutingTable{
routes: make(map[RouteKey]*Route),
byPrefix: make(map[uuid.UUID]map[RouteKey]*Route),
byOriginASN: make(map[uuid.UUID]map[RouteKey]*Route),
byPeerASN: make(map[int]map[RouteKey]*Route),
lastMetricsReset: time.Now(),
snapshotDir: cfg.GetStateDir(),
}
// Try to load from snapshot
if err := rt.loadFromSnapshot(logger); err != nil {
logger.Warn("Failed to load routing table from snapshot", "error", err)
}
return rt
}
// AddRoute adds or updates a route in the routing table
func (rt *RoutingTable) AddRoute(route *Route) {
rt.mu.Lock()
defer rt.mu.Unlock()
key := RouteKey{
PrefixID: route.PrefixID,
OriginASNID: route.OriginASNID,
PeerASN: route.PeerASN,
}
// If route already exists, remove it from indexes first
if existingRoute, exists := rt.routes[key]; exists {
rt.removeFromIndexes(key, existingRoute)
// Decrement counter for existing route
if isIPv6(existingRoute.Prefix) {
rt.ipv6Routes--
} else {
rt.ipv4Routes--
}
}
// Set AddedAt if not already set
if route.AddedAt.IsZero() {
route.AddedAt = time.Now().UTC()
}
// Add to main map
rt.routes[key] = route
// Update indexes
rt.addToIndexes(key, route)
// Update metrics
if isIPv6(route.Prefix) {
rt.ipv6Routes++
atomic.AddUint64(&rt.ipv6Updates, 1)
} else {
rt.ipv4Routes++
atomic.AddUint64(&rt.ipv4Updates, 1)
}
}
// RemoveRoute removes a route from the routing table
func (rt *RoutingTable) RemoveRoute(prefixID, originASNID uuid.UUID, peerASN int) bool {
rt.mu.Lock()
defer rt.mu.Unlock()
key := RouteKey{
PrefixID: prefixID,
OriginASNID: originASNID,
PeerASN: peerASN,
}
route, exists := rt.routes[key]
if !exists {
return false
}
// Remove from indexes
rt.removeFromIndexes(key, route)
// Remove from main map
delete(rt.routes, key)
// Update metrics
if isIPv6(route.Prefix) {
rt.ipv6Routes--
atomic.AddUint64(&rt.ipv6Updates, 1)
} else {
rt.ipv4Routes--
atomic.AddUint64(&rt.ipv4Updates, 1)
}
return true
}
// WithdrawRoutesByPrefixAndPeer removes all routes for a specific prefix from a specific peer
func (rt *RoutingTable) WithdrawRoutesByPrefixAndPeer(prefixID uuid.UUID, peerASN int) int {
rt.mu.Lock()
defer rt.mu.Unlock()
prefixRoutes, exists := rt.byPrefix[prefixID]
if !exists {
return 0
}
// Collect keys to delete (can't delete while iterating)
var keysToDelete []RouteKey
for key, route := range prefixRoutes {
if route.PeerASN == peerASN {
keysToDelete = append(keysToDelete, key)
}
}
// Delete the routes
count := 0
for _, key := range keysToDelete {
route, exists := rt.routes[key]
if !exists {
continue
}
rt.removeFromIndexes(key, route)
delete(rt.routes, key)
count++
// Update metrics
if isIPv6(route.Prefix) {
rt.ipv6Routes--
atomic.AddUint64(&rt.ipv6Updates, 1)
} else {
rt.ipv4Routes--
atomic.AddUint64(&rt.ipv4Updates, 1)
}
}
return count
}
// GetRoute retrieves a specific route
func (rt *RoutingTable) GetRoute(prefixID, originASNID uuid.UUID, peerASN int) (*Route, bool) {
rt.mu.RLock()
defer rt.mu.RUnlock()
key := RouteKey{
PrefixID: prefixID,
OriginASNID: originASNID,
PeerASN: peerASN,
}
route, exists := rt.routes[key]
if !exists {
return nil, false
}
// Return a copy to prevent external modification
routeCopy := *route
return &routeCopy, true
}
// GetRoutesByPrefix returns all routes for a specific prefix
func (rt *RoutingTable) GetRoutesByPrefix(prefixID uuid.UUID) []*Route {
rt.mu.RLock()
defer rt.mu.RUnlock()
routes := make([]*Route, 0)
if prefixRoutes, exists := rt.byPrefix[prefixID]; exists {
for _, route := range prefixRoutes {
routeCopy := *route
routes = append(routes, &routeCopy)
}
}
return routes
}
// GetRoutesByOriginASN returns all routes originated by a specific ASN
func (rt *RoutingTable) GetRoutesByOriginASN(originASNID uuid.UUID) []*Route {
rt.mu.RLock()
defer rt.mu.RUnlock()
routes := make([]*Route, 0)
if asnRoutes, exists := rt.byOriginASN[originASNID]; exists {
for _, route := range asnRoutes {
routeCopy := *route
routes = append(routes, &routeCopy)
}
}
return routes
}
// GetRoutesByPeerASN returns all routes received from a specific peer ASN
func (rt *RoutingTable) GetRoutesByPeerASN(peerASN int) []*Route {
rt.mu.RLock()
defer rt.mu.RUnlock()
routes := make([]*Route, 0)
if peerRoutes, exists := rt.byPeerASN[peerASN]; exists {
for _, route := range peerRoutes {
routeCopy := *route
routes = append(routes, &routeCopy)
}
}
return routes
}
// GetAllRoutes returns all active routes in the routing table
func (rt *RoutingTable) GetAllRoutes() []*Route {
rt.mu.RLock()
defer rt.mu.RUnlock()
routes := make([]*Route, 0, len(rt.routes))
for _, route := range rt.routes {
routeCopy := *route
routes = append(routes, &routeCopy)
}
return routes
}
// Size returns the total number of routes in the table
func (rt *RoutingTable) Size() int {
rt.mu.RLock()
defer rt.mu.RUnlock()
return len(rt.routes)
}
// Stats returns statistics about the routing table
func (rt *RoutingTable) Stats() map[string]int {
rt.mu.RLock()
defer rt.mu.RUnlock()
stats := map[string]int{
"total_routes": len(rt.routes),
"unique_prefixes": len(rt.byPrefix),
"unique_origins": len(rt.byOriginASN),
"unique_peers": len(rt.byPeerASN),
}
return stats
}
// DetailedStats contains detailed routing table statistics
type DetailedStats struct {
IPv4Routes int
IPv6Routes int
IPv4UpdatesRate float64
IPv6UpdatesRate float64
TotalRoutes int
UniquePrefixes int
UniqueOrigins int
UniquePeers int
}
// GetDetailedStats returns detailed statistics including IPv4/IPv6 breakdown and update rates
func (rt *RoutingTable) GetDetailedStats() DetailedStats {
rt.mu.Lock()
defer rt.mu.Unlock()
// Calculate update rates
elapsed := time.Since(rt.lastMetricsReset).Seconds()
ipv4Updates := atomic.LoadUint64(&rt.ipv4Updates)
ipv6Updates := atomic.LoadUint64(&rt.ipv6Updates)
stats := DetailedStats{
IPv4Routes: rt.ipv4Routes,
IPv6Routes: rt.ipv6Routes,
IPv4UpdatesRate: float64(ipv4Updates) / elapsed,
IPv6UpdatesRate: float64(ipv6Updates) / elapsed,
TotalRoutes: len(rt.routes),
UniquePrefixes: len(rt.byPrefix),
UniqueOrigins: len(rt.byOriginASN),
UniquePeers: len(rt.byPeerASN),
}
// Reset counters for next period
atomic.StoreUint64(&rt.ipv4Updates, 0)
atomic.StoreUint64(&rt.ipv6Updates, 0)
rt.lastMetricsReset = time.Now()
return stats
}
// Clear removes all routes from the routing table
func (rt *RoutingTable) Clear() {
rt.mu.Lock()
defer rt.mu.Unlock()
rt.routes = make(map[RouteKey]*Route)
rt.byPrefix = make(map[uuid.UUID]map[RouteKey]*Route)
rt.byOriginASN = make(map[uuid.UUID]map[RouteKey]*Route)
rt.byPeerASN = make(map[int]map[RouteKey]*Route)
rt.ipv4Routes = 0
rt.ipv6Routes = 0
atomic.StoreUint64(&rt.ipv4Updates, 0)
atomic.StoreUint64(&rt.ipv6Updates, 0)
rt.lastMetricsReset = time.Now()
}
// RLock acquires a read lock on the routing table
// This is exposed for the snapshotter to use
func (rt *RoutingTable) RLock() {
rt.mu.RLock()
}
// RUnlock releases a read lock on the routing table
// This is exposed for the snapshotter to use
func (rt *RoutingTable) RUnlock() {
rt.mu.RUnlock()
}
// GetAllRoutesUnsafe returns all routes without copying
// IMPORTANT: Caller must hold RLock before calling this method
func (rt *RoutingTable) GetAllRoutesUnsafe() []*Route {
routes := make([]*Route, 0, len(rt.routes))
for _, route := range rt.routes {
routes = append(routes, route)
}
return routes
}
// Helper methods for index management
func (rt *RoutingTable) addToIndexes(key RouteKey, route *Route) {
// Add to prefix index
if rt.byPrefix[route.PrefixID] == nil {
rt.byPrefix[route.PrefixID] = make(map[RouteKey]*Route)
}
rt.byPrefix[route.PrefixID][key] = route
// Add to origin ASN index
if rt.byOriginASN[route.OriginASNID] == nil {
rt.byOriginASN[route.OriginASNID] = make(map[RouteKey]*Route)
}
rt.byOriginASN[route.OriginASNID][key] = route
// Add to peer ASN index
if rt.byPeerASN[route.PeerASN] == nil {
rt.byPeerASN[route.PeerASN] = make(map[RouteKey]*Route)
}
rt.byPeerASN[route.PeerASN][key] = route
}
func (rt *RoutingTable) removeFromIndexes(key RouteKey, route *Route) {
// Remove from prefix index
if prefixRoutes, exists := rt.byPrefix[route.PrefixID]; exists {
delete(prefixRoutes, key)
if len(prefixRoutes) == 0 {
delete(rt.byPrefix, route.PrefixID)
}
}
// Remove from origin ASN index
if asnRoutes, exists := rt.byOriginASN[route.OriginASNID]; exists {
delete(asnRoutes, key)
if len(asnRoutes) == 0 {
delete(rt.byOriginASN, route.OriginASNID)
}
}
// Remove from peer ASN index
if peerRoutes, exists := rt.byPeerASN[route.PeerASN]; exists {
delete(peerRoutes, key)
if len(peerRoutes) == 0 {
delete(rt.byPeerASN, route.PeerASN)
}
}
}
// String returns a string representation of the route key
func (k RouteKey) String() string {
return fmt.Sprintf("%s/%s/%d", k.PrefixID, k.OriginASNID, k.PeerASN)
}
// isIPv6 returns true if the prefix is an IPv6 address
func isIPv6(prefix string) bool {
return strings.Contains(prefix, ":")
}
// loadFromSnapshot attempts to load the routing table from a snapshot file
func (rt *RoutingTable) loadFromSnapshot(logger *slog.Logger) error {
// If no snapshot directory specified, nothing to load
if rt.snapshotDir == "" {
return nil
}
snapshotPath := filepath.Join(rt.snapshotDir, snapshotFilename)
// Check if snapshot file exists
if _, err := os.Stat(snapshotPath); os.IsNotExist(err) {
// No snapshot file exists, this is normal - start with empty routing table
return nil
}
// Open the snapshot file
file, err := os.Open(filepath.Clean(snapshotPath))
if err != nil {
return fmt.Errorf("failed to open snapshot file: %w", err)
}
defer func() { _ = file.Close() }()
// Create gzip reader
gzReader, err := gzip.NewReader(file)
if err != nil {
return fmt.Errorf("failed to create gzip reader: %w", err)
}
defer func() { _ = gzReader.Close() }()
// Decode the snapshot
var snapshot struct {
Timestamp time.Time `json:"timestamp"`
Stats DetailedStats `json:"stats"`
Routes []*Route `json:"routes"`
}
decoder := json.NewDecoder(gzReader)
if err := decoder.Decode(&snapshot); err != nil {
return fmt.Errorf("failed to decode snapshot: %w", err)
}
// Calculate staleness cutoff time
now := time.Now().UTC()
cutoffTime := now.Add(-routeStalenessThreshold)
// Load non-stale routes
loadedCount := 0
staleCount := 0
for _, route := range snapshot.Routes {
// Check if route is stale based on AddedAt time
if route.AddedAt.Before(cutoffTime) {
staleCount++
continue
}
// Add the route (this will update counters and indexes)
rt.AddRoute(route)
loadedCount++
}
logger.Info("Loaded routing table from snapshot",
"snapshot_time", snapshot.Timestamp,
"loaded_routes", loadedCount,
"stale_routes", staleCount,
"total_routes_in_snapshot", len(snapshot.Routes),
)
return nil
}
internal/routingtable/routingtable_test.go
-245
View File
@@ -1,245 +0,0 @@
package routingtable
import (
"log/slog"
"sync"
"testing"
"time"
"git.eeqj.de/sneak/routewatch/internal/config"
"github.com/google/uuid"
)
func TestRoutingTable(t *testing.T) {
// Create a test logger
logger := slog.Default()
// Create test config with empty state dir (no snapshot loading)
cfg := &config.Config{
StateDir: "",
}
rt := New(cfg, logger)
// Test data
prefixID1 := uuid.New()
prefixID2 := uuid.New()
originASNID1 := uuid.New()
originASNID2 := uuid.New()
route1 := &Route{
PrefixID: prefixID1,
Prefix: "10.0.0.0/8",
OriginASNID: originASNID1,
OriginASN: 64512,
PeerASN: 64513,
ASPath: []int{64513, 64512},
NextHop: "192.168.1.1",
AnnouncedAt: time.Now(),
}
route2 := &Route{
PrefixID: prefixID2,
Prefix: "192.168.0.0/16",
OriginASNID: originASNID2,
OriginASN: 64514,
PeerASN: 64513,
ASPath: []int{64513, 64514},
NextHop: "192.168.1.1",
AnnouncedAt: time.Now(),
}
// Test AddRoute
rt.AddRoute(route1)
rt.AddRoute(route2)
if rt.Size() != 2 {
t.Errorf("Expected 2 routes, got %d", rt.Size())
}
// Test GetRoute
retrievedRoute, exists := rt.GetRoute(prefixID1, originASNID1, 64513)
if !exists {
t.Error("Route 1 should exist")
}
if retrievedRoute.Prefix != "10.0.0.0/8" {
t.Errorf("Expected prefix 10.0.0.0/8, got %s", retrievedRoute.Prefix)
}
// Test GetRoutesByPrefix
prefixRoutes := rt.GetRoutesByPrefix(prefixID1)
if len(prefixRoutes) != 1 {
t.Errorf("Expected 1 route for prefix, got %d", len(prefixRoutes))
}
// Test GetRoutesByPeerASN
peerRoutes := rt.GetRoutesByPeerASN(64513)
if len(peerRoutes) != 2 {
t.Errorf("Expected 2 routes from peer 64513, got %d", len(peerRoutes))
}
// Test RemoveRoute
removed := rt.RemoveRoute(prefixID1, originASNID1, 64513)
if !removed {
t.Error("Route should have been removed")
}
if rt.Size() != 1 {
t.Errorf("Expected 1 route after removal, got %d", rt.Size())
}
// Test WithdrawRoutesByPrefixAndPeer
// Add the route back first
rt.AddRoute(route1)
// Add another route for the same prefix from the same peer
route3 := &Route{
PrefixID: prefixID1,
Prefix: "10.0.0.0/8",
OriginASNID: originASNID2, // Different origin
OriginASN: 64515,
PeerASN: 64513,
ASPath: []int{64513, 64515},
NextHop: "192.168.1.1",
AnnouncedAt: time.Now(),
}
rt.AddRoute(route3)
count := rt.WithdrawRoutesByPrefixAndPeer(prefixID1, 64513)
if count != 2 {
t.Errorf("Expected to withdraw 2 routes, withdrew %d", count)
}
// Should only have route2 left
if rt.Size() != 1 {
t.Errorf("Expected 1 route after withdrawal, got %d", rt.Size())
}
// Test Stats
stats := rt.Stats()
if stats["total_routes"] != 1 {
t.Errorf("Expected 1 total route in stats, got %d", stats["total_routes"])
}
// Test Clear
rt.Clear()
if rt.Size() != 0 {
t.Errorf("Expected 0 routes after clear, got %d", rt.Size())
}
}
func TestRoutingTableConcurrency(t *testing.T) {
// Create a test logger
logger := slog.Default()
// Create test config with empty state dir (no snapshot loading)
cfg := &config.Config{
StateDir: "",
}
rt := New(cfg, logger)
// Test concurrent access
var wg sync.WaitGroup
numGoroutines := 10
numOperations := 100
// Start multiple goroutines that add/remove routes
for i := 0; i < numGoroutines; i++ {
wg.Add(1)
go func(id int) {
defer wg.Done()
for j := 0; j < numOperations; j++ {
prefixID := uuid.New()
originASNID := uuid.New()
route := &Route{
PrefixID: prefixID,
Prefix: "10.0.0.0/8",
OriginASNID: originASNID,
OriginASN: 64512 + id,
PeerASN: 64500,
ASPath: []int{64500, 64512 + id},
NextHop: "192.168.1.1",
AnnouncedAt: time.Now(),
}
// Add route
rt.AddRoute(route)
// Try to get it
_, _ = rt.GetRoute(prefixID, originASNID, 64500)
// Get stats
_ = rt.Stats()
// Remove it
rt.RemoveRoute(prefixID, originASNID, 64500)
}
}(i)
}
wg.Wait()
// Table should be empty after all operations
if rt.Size() != 0 {
t.Errorf("Expected empty table after concurrent operations, got %d routes", rt.Size())
}
}
func TestRouteUpdate(t *testing.T) {
// Create a test logger
logger := slog.Default()
// Create test config with empty state dir (no snapshot loading)
cfg := &config.Config{
StateDir: "",
}
rt := New(cfg, logger)
prefixID := uuid.New()
originASNID := uuid.New()
route1 := &Route{
PrefixID: prefixID,
Prefix: "10.0.0.0/8",
OriginASNID: originASNID,
OriginASN: 64512,
PeerASN: 64513,
ASPath: []int{64513, 64512},
NextHop: "192.168.1.1",
AnnouncedAt: time.Now(),
}
// Add initial route
rt.AddRoute(route1)
// Update the same route with new next hop
route2 := &Route{
PrefixID: prefixID,
Prefix: "10.0.0.0/8",
OriginASNID: originASNID,
OriginASN: 64512,
PeerASN: 64513,
ASPath: []int{64513, 64512},
NextHop: "192.168.1.2", // Changed
AnnouncedAt: time.Now().Add(1 * time.Minute),
}
rt.AddRoute(route2)
// Should still have only 1 route
if rt.Size() != 1 {
t.Errorf("Expected 1 route after update, got %d", rt.Size())
}
// Check that the route was updated
retrievedRoute, exists := rt.GetRoute(prefixID, originASNID, 64513)
if !exists {
t.Error("Route should exist after update")
}
if retrievedRoute.NextHop != "192.168.1.2" {
t.Errorf("Expected updated next hop 192.168.1.2, got %s", retrievedRoute.NextHop)
}
}
internal/server/handlers.go
+1307
View File
File diff suppressed because it is too large Load Diff
internal/server/handlers_internal_test.go
+358
View File
@@ -0,0 +1,358 @@
package server
import (
"context"
"encoding/json"
"errors"
"html"
"net/http"
"net/http/httptest"
"runtime"
"slices"
"strings"
"testing"
"time"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/streamer"
"git.eeqj.de/sneak/routewatch/pkg/asinfo"
"github.com/google/uuid"
)
// The prefix and peer addresses the prefix page tests store routes for.
const (
testPrefixV4 = "192.0.2.0/24"
testPeerV4 = "198.51.100.1"
testPeerV6 = "2001:db8::1"
)
// blockingStatsDB embeds database.Store (left nil) and overrides only
// GetStatsContext, which blocks until release is closed. The stats handlers
// call it in a goroutine; every other Store method is unused on the timeout
// path and would panic if called.
type blockingStatsDB struct {
database.Store
release chan struct{}
}
func (d blockingStatsDB) GetStatsContext(_ context.Context) (database.Stats, error) {
<-d.release
return database.Stats{}, nil
}
// TestStatsHandlersDoNotLeakOnTimeout drives each stats handler repeatedly with
// a request whose context times out before the database responds, then releases
// the blocked queries and asserts the goroutine count returns to its starting
// value. Before the fix the per-request goroutine sent on an unbuffered channel
// that nothing received once the timeout won, so it blocked forever and every
// poll leaked one goroutine.
//
//nolint:paralleltest // counts goroutines, so no other test may run alongside
func TestStatsHandlersDoNotLeakOnTimeout(t *testing.T) {
release := make(chan struct{})
db := blockingStatsDB{release: release}
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
handlers := map[string]http.HandlerFunc{
"status.json": s.handleStatusJSON(),
"stats": s.handleStats(),
}
baseline := settledGoroutineCount()
const (
iterations = 20
requestTimeout = 50 * time.Millisecond
)
for _, handler := range handlers {
for range iterations {
ctx, cancel := context.WithTimeout(context.Background(), requestTimeout)
req := httptest.NewRequestWithContext(ctx, http.MethodGet, "/", nil)
handler(httptest.NewRecorder(), req)
cancel()
}
}
// Let the blocked queries finish; with buffered channels each goroutine's
// send now succeeds and the goroutine exits.
close(release)
if !waitForGoroutines(baseline) {
t.Fatalf("goroutines did not return to baseline %d, got %d",
baseline, runtime.NumGoroutine())
}
}
// TestStatsHandlersAnswerFromMemory checks that both stats handlers answer 200
// with the live route counts and the prefix distribution while the database is
// closed, so that any query would fail: the request path reads them from
// memory. The prefix distribution query it used to run read every live route
// and, on a large database, took the whole 4-second deadline, so
// /api/v1/stats answered 500 (https://git.eeqj.de/sneak/routewatch/issues/30).
// The oldest and newest route times still come from one-row lookups at the ends
// of an index; with the database closed they are left out of the answer.
func TestStatsHandlersAnswerFromMemory(t *testing.T) {
t.Parallel()
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("database.New: %v", err)
}
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
err = db.UpsertLiveRouteBatch([]*database.LiveRoute{
{
ID: uuid.New(), Prefix: "198.51.100.0/24", MaskLength: 24, IPVersion: 4,
OriginASN: 64500, PeerIP: "192.0.2.1", ASPath: []int{64500},
NextHop: "192.0.2.1", LastUpdated: ts,
},
{
ID: uuid.New(), Prefix: "2001:db8::/32", MaskLength: 32, IPVersion: 6,
OriginASN: 64501, PeerIP: testPeerV6, ASPath: []int{64501},
NextHop: testPeerV6, LastUpdated: ts,
},
})
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
err = db.Close()
if err != nil {
t.Fatalf("Close: %v", err)
}
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
handlers := map[string]http.HandlerFunc{
"status.json": s.handleStatusJSON(),
"stats": s.handleStats(),
}
for name, handler := range handlers {
rec := httptest.NewRecorder()
handler(rec, httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil))
if rec.Code != http.StatusOK {
t.Errorf("%s: status %d, want %d; body %s",
name, rec.Code, http.StatusOK, rec.Body)
continue
}
//nolint:tagliatelle // field names are the existing JSON API
var body struct {
Data struct {
IPv4Routes int `json:"ipv4_routes"`
IPv6Routes int `json:"ipv6_routes"`
IPv4Distribution []database.PrefixDistribution `json:"ipv4_prefix_distribution"`
IPv6Distribution []database.PrefixDistribution `json:"ipv6_prefix_distribution"`
} `json:"data"`
}
err := json.Unmarshal(rec.Body.Bytes(), &body)
if err != nil {
t.Fatalf("%s: decoding the answer: %v", name, err)
}
if body.Data.IPv4Routes != 1 || body.Data.IPv6Routes != 1 {
t.Errorf("%s: routes = (v4 %d, v6 %d), want (1, 1)",
name, body.Data.IPv4Routes, body.Data.IPv6Routes)
}
wantV4 := []database.PrefixDistribution{{MaskLength: 24, Count: 1}}
if !slices.Equal(body.Data.IPv4Distribution, wantV4) {
t.Errorf("%s: IPv4 distribution = %v, want %v",
name, body.Data.IPv4Distribution, wantV4)
}
wantV6 := []database.PrefixDistribution{{MaskLength: 32, Count: 1}}
if !slices.Equal(body.Data.IPv6Distribution, wantV6) {
t.Errorf("%s: IPv6 distribution = %v, want %v",
name, body.Data.IPv6Distribution, wantV6)
}
}
}
// TestPrefixDetailHandlersParsePrefix checks that both prefix pages find a
// stored route whatever valid text form the URL writes its prefix in, and
// answer 400 for a prefix that does not parse. The tables store the form
// net/netip prints. Before the fix the URL text was looked up as typed, so
// "2001:DB8:0:0::/32" was not found although "2001:db8::/32" was stored, and
// "foo/24" answered 500 (https://git.eeqj.de/sneak/routewatch/issues/71).
func TestPrefixDetailHandlersParsePrefix(t *testing.T) {
t.Parallel()
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("database.New: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
err = db.UpsertLiveRouteBatch([]*database.LiveRoute{
{
ID: uuid.New(), Prefix: testPrefixV4, MaskLength: 24, IPVersion: 4,
OriginASN: 64500, PeerIP: testPeerV4, ASPath: []int{64500},
NextHop: testPeerV4, LastUpdated: ts,
},
{
ID: uuid.New(), Prefix: "2001:db8::/32", MaskLength: 32, IPVersion: 6,
OriginASN: 64501, PeerIP: testPeerV6, ASPath: []int{64501},
NextHop: testPeerV6, LastUpdated: ts,
},
})
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
s := New(db, streamer.New(logger.New(), metrics.New()), logger.New(), &config.Config{})
tests := []struct {
path string
want int
}{
{"/api/v1/prefix/foo/24", http.StatusBadRequest},
{"/prefix/foo/24", http.StatusBadRequest},
{"/api/v1/prefix/2001:DB8:0:0::/32", http.StatusOK},
{"/prefix/2001:DB8:0:0::/32", http.StatusOK},
{"/api/v1/prefix/192.0.2.1/24", http.StatusOK},
{"/prefix/192.0.2.1/24", http.StatusOK},
}
for _, tt := range tests {
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, tt.path, nil))
if rec.Code != tt.want {
t.Errorf("%s: status %d, want %d; body %s",
tt.path, rec.Code, tt.want, rec.Body)
}
}
}
// errASDetailsLookupFailed is what failingASDetailsDB answers every AS details
// lookup with.
var errASDetailsLookupFailed = errors.New("AS details lookup failed")
// failingASDetailsDB wraps a real store and fails every AS details lookup.
type failingASDetailsDB struct {
database.Store
}
func (failingASDetailsDB) GetASDetailsContext(
_ context.Context, _ int,
) (*database.ASN, []database.LiveRoute, error) {
return nil, nil, errASDetailsLookupFailed
}
// TestPrefixDetailShowsOriginNames checks that the prefix page shows each
// origin AS's handle and description without looking up the AS's details,
// which also read every route the AS originates. Before the fix the page did
// that lookup once per origin, so with it failing the names were missing
// (https://git.eeqj.de/sneak/routewatch/issues/67). The AS path column shows
// handles too, so the test looks for the origin list's "(handle)" form.
func TestPrefixDetailShowsOriginNames(t *testing.T) {
t.Parallel()
db, err := database.New(&config.Config{StateDir: t.TempDir()}, logger.New())
if err != nil {
t.Fatalf("database.New: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
ts := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
origins := []int{15169, 13335}
for _, asn := range origins {
_, err := db.GetOrCreateASN(asn, ts)
if err != nil {
t.Fatalf("GetOrCreateASN(%d): %v", asn, err)
}
}
err = db.UpsertLiveRouteBatch([]*database.LiveRoute{
{
ID: uuid.New(), Prefix: testPrefixV4, MaskLength: 24, IPVersion: 4,
OriginASN: origins[0], PeerIP: testPeerV4,
ASPath: []int{origins[0]}, NextHop: testPeerV4, LastUpdated: ts,
},
{
ID: uuid.New(), Prefix: testPrefixV4, MaskLength: 24, IPVersion: 4,
OriginASN: origins[1], PeerIP: "198.51.100.2",
ASPath: []int{origins[1]}, NextHop: "198.51.100.2", LastUpdated: ts,
},
})
if err != nil {
t.Fatalf("UpsertLiveRouteBatch: %v", err)
}
s := New(failingASDetailsDB{db}, streamer.New(logger.New(), metrics.New()),
logger.New(), &config.Config{})
rec := httptest.NewRecorder()
s.router.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, "/prefix/"+testPrefixV4, nil))
if rec.Code != http.StatusOK {
t.Fatalf("status %d, want %d; body %s", rec.Code, http.StatusOK, rec.Body)
}
page := rec.Body.String()
for _, asn := range origins {
info, ok := asinfo.Get(asn)
if !ok {
t.Fatalf("AS%d is not in the AS info data", asn)
}
want := "(" + html.EscapeString(info.Handle) + ")"
if !strings.Contains(page, want) {
t.Errorf("page does not show AS%d's handle %q", asn, want)
}
want = html.EscapeString(info.Description)
if !strings.Contains(page, want) {
t.Errorf("page does not show AS%d's description %q", asn, want)
}
}
}
// settledGoroutineCount lets transient goroutines finish, then reports the
// current count.
func settledGoroutineCount() int {
prev := runtime.NumGoroutine()
for range 20 {
time.Sleep(10 * time.Millisecond)
cur := runtime.NumGoroutine()
if cur == prev {
return cur
}
prev = cur
}
return prev
}
// waitForGoroutines waits until the goroutine count drops to target or below.
func waitForGoroutines(target int) bool {
for range 100 {
if runtime.NumGoroutine() <= target {
return true
}
time.Sleep(10 * time.Millisecond)
}
return false
}
internal/server/middleware.go
+210 -13
View File
@@ -5,6 +5,7 @@ import (
"context"
"encoding/json"
"fmt"
"log/slog"
"net/http"
"sync"
"time"
@@ -13,6 +14,7 @@ import (
// responseWriter wraps http.ResponseWriter to capture the response
type responseWriter struct {
http.ResponseWriter
body *bytes.Buffer
statusCode int
written bool
@@ -44,7 +46,12 @@ func (rw *responseWriter) Header() http.Header {
return rw.ResponseWriter.Header()
}
// JSONResponseMiddleware wraps all JSON responses with metadata
// JSONResponseMiddleware is an HTTP middleware that wraps all JSON responses
// with a @meta field containing execution metadata. The metadata includes the
// time zone (always UTC), API version, and request execution time in milliseconds.
//
// Endpoints "/" and "/status" are excluded from this processing and passed through
// unchanged. Non-JSON responses and empty responses are also passed through unchanged.
func JSONResponseMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Skip non-JSON endpoints
@@ -80,8 +87,10 @@ func JSONResponseMiddleware(next http.Handler) http.Handler {
}
// Parse the original response
var originalResponse map[string]interface{}
if err := json.Unmarshal(rw.body.Bytes(), &originalResponse); err != nil {
var originalResponse map[string]any
err := json.Unmarshal(rw.body.Bytes(), &originalResponse)
if err != nil {
// If we can't parse it, just write original
w.WriteHeader(rw.statusCode)
_, _ = w.Write(rw.body.Bytes())
@@ -90,7 +99,7 @@ func JSONResponseMiddleware(next http.Handler) http.Handler {
}
// Add @meta field
originalResponse["@meta"] = map[string]interface{}{
originalResponse["@meta"] = map[string]any{
"time_zone": "UTC",
"api_version": 1,
"execution_time": fmt.Sprintf("%d ms", execTime.Milliseconds()),
@@ -106,8 +115,10 @@ func JSONResponseMiddleware(next http.Handler) http.Handler {
// timeoutWriter wraps ResponseWriter to prevent concurrent writes after timeout
type timeoutWriter struct {
http.ResponseWriter
mu sync.Mutex
written bool
header http.Header // cached header to prevent concurrent access
}
func (tw *timeoutWriter) Write(b []byte) (int, error) {
@@ -133,17 +144,39 @@ func (tw *timeoutWriter) WriteHeader(statusCode int) {
}
func (tw *timeoutWriter) Header() http.Header {
tw.mu.Lock()
defer tw.mu.Unlock()
if tw.written {
// Return a copy to prevent modifications after timeout
if tw.header == nil {
tw.header = make(http.Header)
}
return tw.header
}
return tw.ResponseWriter.Header()
}
func (tw *timeoutWriter) markWritten() {
tw.mu.Lock()
defer tw.mu.Unlock()
tw.written = true
}
// TimeoutMiddleware creates a timeout middleware that returns JSON errors
func TimeoutMiddleware(timeout time.Duration) func(http.Handler) http.Handler {
// TimeoutMiddleware creates an HTTP middleware that enforces a request timeout.
// If the handler does not complete within the specified duration, the middleware
// returns a JSON error response with HTTP status 408 (Request Timeout).
//
// The timeout parameter specifies the maximum duration allowed for request processing.
// The returned middleware handles panics from the wrapped handler by re-panicking
// after cleanup, and prevents concurrent writes to the response after timeout occurs.
func TimeoutMiddleware(
timeout time.Duration,
logger *slog.Logger,
) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
startTime := time.Now()
@@ -153,10 +186,11 @@ func TimeoutMiddleware(timeout time.Duration) func(http.Handler) http.Handler {
tw := &timeoutWriter{
ResponseWriter: w,
header: make(http.Header),
}
done := make(chan struct{})
panicChan := make(chan interface{}, 1)
panicChan := make(chan any, 1)
go func() {
defer func() {
@@ -176,18 +210,31 @@ func TimeoutMiddleware(timeout time.Duration) func(http.Handler) http.Handler {
return
case <-ctx.Done():
tw.markWritten() // Prevent the handler from writing after timeout
execTime := time.Since(startTime)
// Log the timeout as a warning
logger.Warn("Request timeout",
"method", r.Method,
"path", r.URL.Path,
"duration_ms", execTime.Milliseconds(),
"remote_addr", r.RemoteAddr,
)
// Write directly to the underlying writer since we've marked tw as written
// This is safe because markWritten() prevents the handler from writing
tw.mu.Lock()
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusRequestTimeout)
tw.mu.Unlock()
response := map[string]interface{}{
"status": "error",
"error": map[string]interface{}{
"msg": "Request timeout",
"code": http.StatusRequestTimeout,
response := map[string]any{
jsonKeyStatus: jsonStatusError,
jsonKeyError: map[string]any{
jsonKeyMsg: "Request timeout",
jsonKeyCode: http.StatusRequestTimeout,
},
"@meta": map[string]interface{}{
"@meta": map[string]any{
"time_zone": "UTC",
"api_version": 1,
"execution_time": fmt.Sprintf("%d ms", execTime.Milliseconds()),
@@ -199,3 +246,153 @@ func TimeoutMiddleware(timeout time.Duration) func(http.Handler) http.Handler {
})
}
}
// JSONValidationMiddleware is an HTTP middleware that validates JSON API responses.
// It ensures that responses with Content-Type "application/json" contain valid JSON.
//
// If a response is not valid JSON or is empty when JSON is expected, the middleware
// returns a properly formatted JSON error response. For timeout errors (status 408),
// the error message will be "Request timeout". For other errors, it returns
// "Internal server error" with status 500 if the original status was 200.
//
// Non-JSON responses are passed through unchanged.
func JSONValidationMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Create a custom response writer to capture the response
rw := &responseWriter{
ResponseWriter: w,
body: &bytes.Buffer{},
statusCode: http.StatusOK,
}
// Serve the request
next.ServeHTTP(rw, r)
// Check if it's meant to be a JSON response
contentType := rw.Header().Get("Content-Type")
isJSON := contentType == "application/json" || contentType == ""
// If it's not JSON or has content, pass through
if !isJSON && rw.body.Len() > 0 {
w.WriteHeader(rw.statusCode)
_, _ = w.Write(rw.body.Bytes())
return
}
// For JSON responses, validate the JSON
if rw.body.Len() > 0 {
var testParse any
err := json.Unmarshal(rw.body.Bytes(), &testParse)
if err == nil {
// Valid JSON, write it out
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(rw.statusCode)
_, _ = w.Write(rw.body.Bytes())
return
}
}
// If we get here, either there's no body or invalid JSON
// Write a proper error response
w.Header().Set("Content-Type", "application/json")
// Determine appropriate status code
statusCode := rw.statusCode
if statusCode == http.StatusOK {
statusCode = http.StatusInternalServerError
}
w.WriteHeader(statusCode)
errorMsg := "Internal server error"
if statusCode == http.StatusRequestTimeout {
errorMsg = "Request timeout"
}
response := map[string]any{
jsonKeyStatus: jsonStatusError,
jsonKeyError: map[string]any{
jsonKeyMsg: errorMsg,
jsonKeyCode: statusCode,
},
}
_ = json.NewEncoder(w).Encode(response)
})
}
// statusWriter wraps http.ResponseWriter to capture the status code
type statusWriter struct {
http.ResponseWriter
statusCode int
written bool
}
func (sw *statusWriter) WriteHeader(statusCode int) {
if !sw.written {
sw.statusCode = statusCode
sw.written = true
}
sw.ResponseWriter.WriteHeader(statusCode)
}
func (sw *statusWriter) Write(b []byte) (int, error) {
if !sw.written {
sw.statusCode = http.StatusOK
sw.written = true
}
return sw.ResponseWriter.Write(b)
}
// RequestLoggerMiddleware creates a structured logging middleware using slog.
func RequestLoggerMiddleware(logger *slog.Logger) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
start := time.Now()
// Wrap response writer to capture status
sw := &statusWriter{ResponseWriter: w, statusCode: http.StatusOK}
// Log request start
logger.Debug("HTTP request started",
"method", r.Method,
"path", r.URL.Path,
"remote_addr", r.RemoteAddr,
"user_agent", r.UserAgent(),
)
// Serve the request
next.ServeHTTP(sw, r)
// Log request completion
duration := time.Since(start)
logLevel := slog.LevelInfo
// Slow query threshold (1 second)
const slowQueryThreshold = 1 * time.Second
switch {
case sw.statusCode >= http.StatusInternalServerError:
logLevel = slog.LevelError
case sw.statusCode >= http.StatusBadRequest:
logLevel = slog.LevelWarn
case duration >= slowQueryThreshold:
logLevel = slog.LevelWarn
}
logger.Log(r.Context(), logLevel, "HTTP request completed",
"method", r.Method,
"path", r.URL.Path,
"status", sw.statusCode,
"duration_ms", duration.Milliseconds(),
"remote_addr", r.RemoteAddr,
"slow", duration >= slowQueryThreshold,
)
})
}
}
internal/server/routes.go
+57
View File
@@ -0,0 +1,57 @@
package server
import (
"time"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
)
// setupRoutes configures the HTTP routes
func (s *Server) setupRoutes() {
r := chi.NewRouter()
// Middleware
r.Use(middleware.RequestID)
r.Use(middleware.RealIP)
r.Use(RequestLoggerMiddleware(s.logger.Logger)) // Structured request logging
r.Use(middleware.Recoverer)
const requestTimeout = 30 * time.Second // Increased from 8s for slow queries
r.Use(TimeoutMiddleware(requestTimeout, s.logger.Logger))
r.Use(JSONResponseMiddleware)
// Routes
r.Get("/", s.handleIndex())
r.Get("/status", s.handleStatusHTML())
r.Get("/status.json", JSONValidationMiddleware(s.handleStatusJSON()).ServeHTTP)
r.Get(
"/.well-known/healthcheck.json",
JSONValidationMiddleware(s.handleHealthCheck()).ServeHTTP,
)
// AS and prefix detail pages
r.Get("/as/{asn}", s.handleASDetail())
r.Get("/prefix/{prefix}/{len}", s.handlePrefixDetail())
r.Get("/prefixlength/{length}", s.handlePrefixLength())
r.Get("/prefixlength6/{length}", s.handlePrefixLength6())
// IP info JSON endpoints (replaces old /ip redirect)
r.Route("/ip", func(r chi.Router) {
r.Use(JSONValidationMiddleware)
r.Get("/", s.handleIPInfo()) // Client IP
r.Get("/{addr}", s.handleIPInfo()) // Specified IP
})
// API routes
r.Route("/api/v1", func(r chi.Router) {
r.Use(JSONValidationMiddleware)
r.Get("/stats", s.handleStats())
r.Get("/ip/{ip}", s.handleIPLookup())
r.Get("/as/{asn}", s.handleASDetailJSON())
r.Get("/prefix/{prefix}/{len}", s.handlePrefixDetailJSON())
})
s.router = r
}
internal/server/server.go
+60 -328
View File
@@ -3,37 +3,57 @@ package server
import (
"context"
"encoding/json"
"log/slog"
"errors"
"fmt"
"net"
"net/http"
"os"
"strconv"
"time"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/database"
"git.eeqj.de/sneak/routewatch/internal/routingtable"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/streamer"
"git.eeqj.de/sneak/routewatch/internal/templates"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
)
// ASNFetcherStats contains WHOIS fetcher statistics.
type ASNFetcherStats struct {
SuccessesLastHour int
ErrorsLastHour int
CurrentInterval time.Duration
ConsecutiveFails int
}
// ASNFetcher is an interface for queuing ASN WHOIS lookups.
type ASNFetcher interface {
QueueImmediate(asn int)
GetStats() ASNFetcherStats
}
// Server provides HTTP endpoints for status monitoring
type Server struct {
router *chi.Mux
db database.Store
routingTable *routingtable.RoutingTable
streamer *streamer.Streamer
logger *slog.Logger
logger *logger.Logger
port int
srv *http.Server
asnFetcher ASNFetcher
}
// New creates a new HTTP server
func New(db database.Store, rt *routingtable.RoutingTable, streamer *streamer.Streamer, logger *slog.Logger) *Server {
func New(
db database.Store,
streamer *streamer.Streamer,
logger *logger.Logger,
cfg *config.Config,
) *Server {
s := &Server{
db: db,
routingTable: rt,
streamer: streamer,
logger: logger,
port: cfg.Port,
}
s.setupRoutes()
@@ -41,50 +61,41 @@ func New(db database.Store, rt *routingtable.RoutingTable, streamer *streamer.St
return s
}
// setupRoutes configures the HTTP routes
func (s *Server) setupRoutes() {
r := chi.NewRouter()
// Start listens on the configured port and serves HTTP in the background.
// It returns an error if it cannot listen, for example because the port is
// already in use. ctx bounds only opening the port, not serving.
func (s *Server) Start(ctx context.Context) error {
const (
network = "tcp"
readHeaderTimeout = 40 * time.Second
readTimeout = 60 * time.Second
writeTimeout = 60 * time.Second
idleTimeout = 120 * time.Second
)
// Middleware
r.Use(middleware.RequestID)
r.Use(middleware.RealIP)
r.Use(middleware.Logger)
r.Use(middleware.Recoverer)
const requestTimeout = 2 * time.Second
r.Use(TimeoutMiddleware(requestTimeout))
r.Use(JSONResponseMiddleware)
// Routes
r.Get("/", s.handleRoot())
r.Get("/status", s.handleStatusHTML())
r.Get("/status.json", s.handleStatusJSON())
// API routes
r.Route("/api/v1", func(r chi.Router) {
r.Get("/stats", s.handleStats())
})
s.router = r
}
// Start starts the HTTP server
func (s *Server) Start() error {
port := os.Getenv("PORT")
if port == "" {
port = "8080"
}
const readHeaderTimeout = 10 * time.Second
s.srv = &http.Server{
Addr: ":" + port,
Addr: ":" + strconv.Itoa(s.port),
Handler: s.router,
ReadHeaderTimeout: readHeaderTimeout,
ReadTimeout: readTimeout,
WriteTimeout: writeTimeout,
IdleTimeout: idleTimeout,
}
s.logger.Info("Starting HTTP server", "port", port)
s.logger.Info("Starting HTTP server", "port", s.port, "addr", s.srv.Addr)
var listenConfig net.ListenConfig
listener, err := listenConfig.Listen(ctx, network, s.srv.Addr)
if err != nil {
return fmt.Errorf("starting HTTP server: %w", err)
}
s.logger.Info("HTTP server listening", "addr", listener.Addr().String())
go func() {
if err := s.srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
err := s.srv.Serve(listener)
if err != nil && !errors.Is(err, http.ErrServerClosed) {
s.logger.Error("HTTP server error", "error", err)
}
}()
@@ -103,286 +114,7 @@ func (s *Server) Stop(ctx context.Context) error {
return s.srv.Shutdown(ctx)
}
// handleRoot returns a handler that redirects to /status
func (s *Server) handleRoot() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/status", http.StatusSeeOther)
}
}
// handleStatusJSON returns a handler that serves JSON statistics
func (s *Server) handleStatusJSON() http.HandlerFunc {
// Stats represents the statistics response
type Stats struct {
Uptime string `json:"uptime"`
TotalMessages uint64 `json:"total_messages"`
TotalBytes uint64 `json:"total_bytes"`
MessagesPerSec float64 `json:"messages_per_sec"`
MbitsPerSec float64 `json:"mbits_per_sec"`
Connected bool `json:"connected"`
ASNs int `json:"asns"`
Prefixes int `json:"prefixes"`
IPv4Prefixes int `json:"ipv4_prefixes"`
IPv6Prefixes int `json:"ipv6_prefixes"`
Peerings int `json:"peerings"`
DatabaseSizeBytes int64 `json:"database_size_bytes"`
LiveRoutes int `json:"live_routes"`
IPv4Routes int `json:"ipv4_routes"`
IPv6Routes int `json:"ipv6_routes"`
IPv4UpdatesPerSec float64 `json:"ipv4_updates_per_sec"`
IPv6UpdatesPerSec float64 `json:"ipv6_updates_per_sec"`
}
return func(w http.ResponseWriter, r *http.Request) {
// Create a 1 second timeout context for this request
ctx, cancel := context.WithTimeout(r.Context(), 1*time.Second)
defer cancel()
metrics := s.streamer.GetMetrics()
// Get database stats with timeout
statsChan := make(chan database.Stats)
errChan := make(chan error)
go func() {
s.logger.Debug("Starting database stats query")
dbStats, err := s.db.GetStats()
if err != nil {
s.logger.Debug("Database stats query failed", "error", err)
errChan <- err
return
}
s.logger.Debug("Database stats query completed")
statsChan <- dbStats
}()
var dbStats database.Stats
select {
case <-ctx.Done():
s.logger.Error("Database stats timeout in status.json")
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusRequestTimeout)
_ = json.NewEncoder(w).Encode(map[string]interface{}{
"status": "error",
"error": map[string]interface{}{
"msg": "Database timeout",
"code": http.StatusRequestTimeout,
},
})
return
case err := <-errChan:
s.logger.Error("Failed to get database stats", "error", err)
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusInternalServerError)
_ = json.NewEncoder(w).Encode(map[string]interface{}{
"status": "error",
"error": map[string]interface{}{
"msg": err.Error(),
"code": http.StatusInternalServerError,
},
})
return
case dbStats = <-statsChan:
// Success
}
uptime := time.Since(metrics.ConnectedSince).Truncate(time.Second).String()
if metrics.ConnectedSince.IsZero() {
uptime = "0s"
}
const bitsPerMegabit = 1000000.0
// Get detailed routing table stats
rtStats := s.routingTable.GetDetailedStats()
stats := Stats{
Uptime: uptime,
TotalMessages: metrics.TotalMessages,
TotalBytes: metrics.TotalBytes,
MessagesPerSec: metrics.MessagesPerSec,
MbitsPerSec: metrics.BitsPerSec / bitsPerMegabit,
Connected: metrics.Connected,
ASNs: dbStats.ASNs,
Prefixes: dbStats.Prefixes,
IPv4Prefixes: dbStats.IPv4Prefixes,
IPv6Prefixes: dbStats.IPv6Prefixes,
Peerings: dbStats.Peerings,
DatabaseSizeBytes: dbStats.FileSizeBytes,
LiveRoutes: rtStats.TotalRoutes,
IPv4Routes: rtStats.IPv4Routes,
IPv6Routes: rtStats.IPv6Routes,
IPv4UpdatesPerSec: rtStats.IPv4UpdatesRate,
IPv6UpdatesPerSec: rtStats.IPv6UpdatesRate,
}
w.Header().Set("Content-Type", "application/json")
response := map[string]interface{}{
"status": "ok",
"data": stats,
}
if err := json.NewEncoder(w).Encode(response); err != nil {
s.logger.Error("Failed to encode stats", "error", err)
}
}
}
// handleStats returns a handler that serves API v1 statistics
func (s *Server) handleStats() http.HandlerFunc {
// HandlerStatsInfo represents handler statistics in the API response
type HandlerStatsInfo struct {
Name string `json:"name"`
QueueLength int `json:"queue_length"`
QueueCapacity int `json:"queue_capacity"`
ProcessedCount uint64 `json:"processed_count"`
DroppedCount uint64 `json:"dropped_count"`
AvgProcessTimeMs float64 `json:"avg_process_time_ms"`
MinProcessTimeMs float64 `json:"min_process_time_ms"`
MaxProcessTimeMs float64 `json:"max_process_time_ms"`
}
// StatsResponse represents the API statistics response
type StatsResponse struct {
Uptime string `json:"uptime"`
TotalMessages uint64 `json:"total_messages"`
TotalBytes uint64 `json:"total_bytes"`
MessagesPerSec float64 `json:"messages_per_sec"`
MbitsPerSec float64 `json:"mbits_per_sec"`
Connected bool `json:"connected"`
ASNs int `json:"asns"`
Prefixes int `json:"prefixes"`
IPv4Prefixes int `json:"ipv4_prefixes"`
IPv6Prefixes int `json:"ipv6_prefixes"`
Peerings int `json:"peerings"`
DatabaseSizeBytes int64 `json:"database_size_bytes"`
LiveRoutes int `json:"live_routes"`
IPv4Routes int `json:"ipv4_routes"`
IPv6Routes int `json:"ipv6_routes"`
IPv4UpdatesPerSec float64 `json:"ipv4_updates_per_sec"`
IPv6UpdatesPerSec float64 `json:"ipv6_updates_per_sec"`
HandlerStats []HandlerStatsInfo `json:"handler_stats"`
}
return func(w http.ResponseWriter, r *http.Request) {
// Create a 1 second timeout context for this request
ctx, cancel := context.WithTimeout(r.Context(), 1*time.Second)
defer cancel()
// Check if context is already cancelled
select {
case <-ctx.Done():
http.Error(w, "Request timeout", http.StatusRequestTimeout)
return
default:
}
metrics := s.streamer.GetMetrics()
// Get database stats with timeout
statsChan := make(chan database.Stats)
errChan := make(chan error)
go func() {
s.logger.Debug("Starting database stats query")
dbStats, err := s.db.GetStats()
if err != nil {
s.logger.Debug("Database stats query failed", "error", err)
errChan <- err
return
}
s.logger.Debug("Database stats query completed")
statsChan <- dbStats
}()
var dbStats database.Stats
select {
case <-ctx.Done():
s.logger.Error("Database stats timeout")
http.Error(w, "Database timeout", http.StatusRequestTimeout)
return
case err := <-errChan:
s.logger.Error("Failed to get database stats", "error", err)
http.Error(w, err.Error(), http.StatusInternalServerError)
return
case dbStats = <-statsChan:
// Success
}
uptime := time.Since(metrics.ConnectedSince).Truncate(time.Second).String()
if metrics.ConnectedSince.IsZero() {
uptime = "0s"
}
const bitsPerMegabit = 1000000.0
// Get detailed routing table stats
rtStats := s.routingTable.GetDetailedStats()
// Get handler stats
handlerStats := s.streamer.GetHandlerStats()
handlerStatsInfo := make([]HandlerStatsInfo, 0, len(handlerStats))
const microsecondsPerMillisecond = 1000.0
for _, hs := range handlerStats {
handlerStatsInfo = append(handlerStatsInfo, HandlerStatsInfo{
Name: hs.Name,
QueueLength: hs.QueueLength,
QueueCapacity: hs.QueueCapacity,
ProcessedCount: hs.ProcessedCount,
DroppedCount: hs.DroppedCount,
AvgProcessTimeMs: float64(hs.AvgProcessTime.Microseconds()) / microsecondsPerMillisecond,
MinProcessTimeMs: float64(hs.MinProcessTime.Microseconds()) / microsecondsPerMillisecond,
MaxProcessTimeMs: float64(hs.MaxProcessTime.Microseconds()) / microsecondsPerMillisecond,
})
}
stats := StatsResponse{
Uptime: uptime,
TotalMessages: metrics.TotalMessages,
TotalBytes: metrics.TotalBytes,
MessagesPerSec: metrics.MessagesPerSec,
MbitsPerSec: metrics.BitsPerSec / bitsPerMegabit,
Connected: metrics.Connected,
ASNs: dbStats.ASNs,
Prefixes: dbStats.Prefixes,
IPv4Prefixes: dbStats.IPv4Prefixes,
IPv6Prefixes: dbStats.IPv6Prefixes,
Peerings: dbStats.Peerings,
DatabaseSizeBytes: dbStats.FileSizeBytes,
LiveRoutes: rtStats.TotalRoutes,
IPv4Routes: rtStats.IPv4Routes,
IPv6Routes: rtStats.IPv6Routes,
IPv4UpdatesPerSec: rtStats.IPv4UpdatesRate,
IPv6UpdatesPerSec: rtStats.IPv6UpdatesRate,
HandlerStats: handlerStatsInfo,
}
w.Header().Set("Content-Type", "application/json")
response := map[string]interface{}{
"status": "ok",
"data": stats,
}
if err := json.NewEncoder(w).Encode(response); err != nil {
s.logger.Error("Failed to encode stats", "error", err)
}
}
}
// handleStatusHTML returns a handler that serves the HTML status page
func (s *Server) handleStatusHTML() http.HandlerFunc {
return func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "text/html; charset=utf-8")
tmpl := templates.StatusTemplate()
if err := tmpl.Execute(w, nil); err != nil {
s.logger.Error("Failed to render template", "error", err)
http.Error(w, "Internal Server Error", http.StatusInternalServerError)
}
}
// SetASNFetcher sets the ASN WHOIS fetcher for on-demand lookups.
func (s *Server) SetASNFetcher(fetcher ASNFetcher) {
s.asnFetcher = fetcher
}
internal/server/server_test.go
+47
View File
@@ -0,0 +1,47 @@
package server_test
import (
"context"
"net"
"testing"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/server"
"git.eeqj.de/sneak/routewatch/internal/streamer"
)
// TestStartFailsWhenPortIsHeld holds a port with a listener of its own and
// checks that Start on that port returns an error. Before the fix Start
// listened in a goroutine, only logged the failure and returned nil, so the
// daemon kept running without its web server.
func TestStartFailsWhenPortIsHeld(t *testing.T) {
t.Parallel()
var listenConfig net.ListenConfig
held, err := listenConfig.Listen(t.Context(), "tcp", ":0")
if err != nil {
t.Fatalf("failed to open the test listener: %v", err)
}
defer func() { _ = held.Close() }()
addr, ok := held.Addr().(*net.TCPAddr)
if !ok {
t.Fatalf("test listener address %v is not a TCP address", held.Addr())
}
port := addr.Port
log := logger.New()
s := server.New(nil, streamer.New(log, metrics.New()), log,
&config.Config{Port: port})
defer func() { _ = s.Stop(context.Background()) }()
err = s.Start(t.Context())
if err == nil {
t.Fatalf("Start returned nil while port %d was held by another listener",
port)
}
}
internal/snapshotter/snapshotter.go
-242
View File
@@ -1,242 +0,0 @@
// Package snapshotter provides functionality for creating periodic and on-demand
// snapshots of the routing table state.
package snapshotter
import (
"compress/gzip"
"context"
"encoding/json"
"fmt"
"log/slog"
"os"
"path/filepath"
"sync"
"time"
"git.eeqj.de/sneak/routewatch/internal/config"
"git.eeqj.de/sneak/routewatch/internal/routingtable"
)
const (
snapshotInterval = 10 * time.Minute
snapshotFilename = "routingtable.json.gz"
tempFileSuffix = ".tmp"
)
// Snapshotter handles periodic and on-demand snapshots of the routing table
type Snapshotter struct {
rt *routingtable.RoutingTable
stateDir string
logger *slog.Logger
ctx context.Context
cancel context.CancelFunc
mu sync.Mutex // Ensures only one snapshot runs at a time
wg sync.WaitGroup
lastSnapshot time.Time
}
// New creates a new Snapshotter instance
func New(rt *routingtable.RoutingTable, cfg *config.Config, logger *slog.Logger) (*Snapshotter, error) {
stateDir := cfg.GetStateDir()
// If state directory is specified, ensure it exists
if stateDir != "" {
const stateDirPerms = 0750
if err := os.MkdirAll(stateDir, stateDirPerms); err != nil {
return nil, fmt.Errorf("failed to create snapshot directory: %w", err)
}
}
s := &Snapshotter{
rt: rt,
stateDir: stateDir,
logger: logger,
}
return s, nil
}
// Start begins the periodic snapshot process
func (s *Snapshotter) Start(ctx context.Context) {
s.mu.Lock()
defer s.mu.Unlock()
if s.ctx != nil {
// Already started
return
}
ctx, cancel := context.WithCancel(ctx)
s.ctx = ctx
s.cancel = cancel
// Start periodic snapshot goroutine
s.wg.Add(1)
go s.periodicSnapshot()
}
// periodicSnapshot runs periodic snapshots
func (s *Snapshotter) periodicSnapshot() {
defer s.wg.Done()
ticker := time.NewTicker(snapshotInterval)
defer ticker.Stop()
// Wait for the first interval before taking any snapshots
for {
select {
case <-s.ctx.Done():
return
case <-ticker.C:
if err := s.TakeSnapshot(); err != nil {
s.logger.Error("Failed to take periodic snapshot", "error", err)
}
}
}
}
// TakeSnapshot creates a snapshot of the current routing table state
func (s *Snapshotter) TakeSnapshot() error {
// Can't take snapshot without a state directory
if s.stateDir == "" {
return nil
}
// Ensure only one snapshot runs at a time
s.mu.Lock()
defer s.mu.Unlock()
start := time.Now()
s.logger.Info("Starting routing table snapshot")
// Get a copy of all routes while holding read lock
copyStart := time.Now()
s.rt.RLock()
routes := s.rt.GetAllRoutesUnsafe() // We'll need to add this method
s.rt.RUnlock()
// Get stats separately to avoid deadlock
stats := s.rt.GetDetailedStats()
s.logger.Info("Copied routes from routing table",
"duration", time.Since(copyStart),
"route_count", len(routes))
// Create snapshot data structure
snapshot := struct {
Timestamp time.Time `json:"timestamp"`
Stats routingtable.DetailedStats `json:"stats"`
Routes []*routingtable.Route `json:"routes"`
}{
Timestamp: time.Now().UTC(),
Stats: stats,
Routes: routes,
}
// Serialize to JSON
marshalStart := time.Now()
jsonData, err := json.Marshal(snapshot)
if err != nil {
return fmt.Errorf("failed to marshal snapshot: %w", err)
}
s.logger.Info("Marshaled snapshot to JSON",
"duration", time.Since(marshalStart),
"size_bytes", len(jsonData))
// Write compressed data to temporary file
tempPath := filepath.Join(s.stateDir, snapshotFilename+tempFileSuffix)
finalPath := filepath.Join(s.stateDir, snapshotFilename)
// Clean the paths to avoid any path traversal issues
tempPath = filepath.Clean(tempPath)
finalPath = filepath.Clean(finalPath)
tempFile, err := os.Create(tempPath)
if err != nil {
return fmt.Errorf("failed to create temporary file: %w", err)
}
defer func() {
_ = tempFile.Close()
// Clean up temp file if it still exists
_ = os.Remove(tempPath)
}()
// Create gzip writer
gzipWriter := gzip.NewWriter(tempFile)
gzipWriter.Comment = fmt.Sprintf("RouteWatch snapshot taken at %s", snapshot.Timestamp.Format(time.RFC3339))
// Write compressed data
writeStart := time.Now()
if _, err := gzipWriter.Write(jsonData); err != nil {
return fmt.Errorf("failed to write compressed data: %w", err)
}
// Close gzip writer to flush all data
if err := gzipWriter.Close(); err != nil {
return fmt.Errorf("failed to close gzip writer: %w", err)
}
// Sync to disk
if err := tempFile.Sync(); err != nil {
return fmt.Errorf("failed to sync temporary file: %w", err)
}
// Close temp file before rename
if err := tempFile.Close(); err != nil {
return fmt.Errorf("failed to close temporary file: %w", err)
}
// Atomically rename temp file to final location
if err := os.Rename(tempPath, finalPath); err != nil {
return fmt.Errorf("failed to rename temporary file: %w", err)
}
s.logger.Info("Wrote compressed snapshot to disk",
"duration", time.Since(writeStart))
duration := time.Since(start)
s.lastSnapshot = time.Now()
s.logger.Info("Routing table snapshot completed",
"duration", duration,
"routes", len(routes),
"ipv4_routes", stats.IPv4Routes,
"ipv6_routes", stats.IPv6Routes,
"size_bytes", len(jsonData),
"path", finalPath,
)
return nil
}
// Shutdown performs a final snapshot and cleans up resources
func (s *Snapshotter) Shutdown() error {
s.logger.Info("Shutting down snapshotter")
// Cancel context to stop periodic snapshots
if s.cancel != nil {
s.cancel()
}
// Wait for periodic snapshot goroutine to finish
s.wg.Wait()
// Take final snapshot
if err := s.TakeSnapshot(); err != nil {
return fmt.Errorf("failed to take final snapshot: %w", err)
}
return nil
}
// GetLastSnapshotTime returns the time of the last successful snapshot
func (s *Snapshotter) GetLastSnapshotTime() time.Time {
s.mu.Lock()
defer s.mu.Unlock()
return s.lastSnapshot
}
// GetSnapshotPath returns the path to the snapshot file
func (s *Snapshotter) GetSnapshotPath() string {
return filepath.Join(s.stateDir, snapshotFilename)
}
internal/streamer/streamer.go
+468 -110
View File
@@ -1,47 +1,99 @@
// Package streamer implements an HTTP client that connects to the RIPE RIS Live streaming API,
// parses BGP UPDATE messages from the JSON stream, and dispatches them to registered handlers.
// Package streamer implements an HTTP client that connects to the RIPE RIS Live
// streaming API, parses BGP UPDATE messages from the JSON stream, and
// dispatches them to registered handlers.
package streamer
import (
"bufio"
"compress/gzip"
"context"
"encoding/json"
"errors"
"fmt"
"log/slog"
"io"
"math"
"math/rand"
"net/http"
"os"
"sync"
"sync/atomic"
"time"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
)
// countingReader wraps an io.Reader and counts bytes read
type countingReader struct {
reader io.Reader
count atomic.Int64
}
// Read implements io.Reader and counts bytes
func (c *countingReader) Read(p []byte) (int, error) {
n, err := c.reader.Read(p)
c.count.Add(int64(n))
return n, err
}
// Count returns the total bytes read
func (c *countingReader) Count() int64 {
return c.count.Load()
}
// Configuration constants for the RIS Live streamer.
const (
risLiveURL = "https://ris-live.ripe.net/v1/stream/?format=json"
risLiveURL = "https://ris-live.ripe.net/v1/stream/?format=json&" +
"client=https%3A%2F%2Fgit.eeqj.de%2Fsneak%2Froutewatch"
metricsWindowSize = 60 // seconds for rolling average
metricsUpdateRate = time.Second
minBackoffDelay = 5 * time.Second
maxBackoffDelay = 320 * time.Second
// Reconnect when no line arrives for this long
streamIdleTimeout = 60 * time.Second
metricsLogInterval = 10 * time.Second
bytesPerKB = 1024
bytesPerMB = 1024 * 1024
maxConcurrentHandlers = 800 // Maximum number of concurrent message handlers
// Backpressure constants
backpressureThreshold = 0.5 // Start dropping at 50% queue utilization
backpressureSlope = 2.0 // Slope for linear drop probability increase
)
// MessageHandler is an interface for handling RIS messages
var (
errAlreadyRunning = errors.New("streamer already running")
errUnexpectedStatus = errors.New("unexpected status code")
// errStreamEnded is returned by stream when RIS Live ends the response.
errStreamEnded = errors.New("RIS Live ended the stream")
// errStreamIdle is returned by stream when no line arrived for the idle
// timeout.
errStreamIdle = errors.New("no line received")
)
// MessageHandler defines the interface for processing RIS messages.
// Implementations must specify which message types they want to receive,
// how to process messages, and their desired queue capacity.
type MessageHandler interface {
// WantsMessage returns true if this handler wants to process messages of the given type
// WantsMessage returns true if this handler wants to process messages of
// the given type.
WantsMessage(messageType string) bool
// HandleMessage processes a RIS message
// HandleMessage processes a RIS message. This method is called from a dedicated
// goroutine for each handler, so implementations do not need to be goroutine-safe
// with respect to other handlers.
HandleMessage(msg *ristypes.RISMessage)
// QueueCapacity returns the desired queue capacity for this handler
// Handlers that process quickly can have larger queues
// QueueCapacity returns the desired queue capacity for this handler.
// Handlers that process quickly can have larger queues to buffer bursts.
// When the queue fills up, messages will be dropped according to the
// backpressure algorithm.
QueueCapacity() int
}
// RawMessageHandler is a callback for handling raw JSON lines from the stream
// RawMessageHandler is a function type for processing raw JSON lines from the stream.
// It receives the unmodified JSON line as a string before any parsing occurs.
type RawMessageHandler func(line string)
// handlerMetrics tracks performance metrics for a handler
@@ -51,6 +103,7 @@ type handlerMetrics struct {
totalTime time.Duration // Total processing time (for average calculation)
minTime time.Duration // Minimum processing time
maxTime time.Duration // Maximum processing time
queueHighWaterMark int // Maximum queue length seen
mu sync.Mutex // Protects the metrics
}
@@ -61,32 +114,58 @@ type handlerInfo struct {
metrics handlerMetrics
}
// Streamer handles streaming BGP updates from RIS Live
// Streamer manages a connection to the RIPE RIS Live streaming API for receiving
// real-time BGP UPDATE messages. It handles automatic reconnection with exponential
// backoff, dispatches messages to registered handlers via per-handler queues, and
// implements backpressure to prevent queue overflow during high traffic periods.
type Streamer struct {
logger *slog.Logger
logger *logger.Logger
client *http.Client
url string
// Drop the connection when no line arrives for this long
idleTimeout time.Duration
// First wait before reconnecting; doubles up to maxBackoffDelay
minBackoff time.Duration
handlers []*handlerInfo
rawHandler RawMessageHandler
mu sync.RWMutex
cancel context.CancelFunc
running bool
metrics *metrics.Tracker
totalDropped uint64 // Total dropped messages across all handlers
totalDropped atomic.Uint64 // Total dropped messages across all handlers
random *rand.Rand // Random number generator for backpressure drops
bgpPeers map[string]bool // Track active BGP peers by peer IP
bgpPeersMu sync.RWMutex // Protects bgpPeers map
}
// New creates a new RIS streamer
func New(logger *slog.Logger, metrics *metrics.Tracker) *Streamer {
// New creates a new Streamer instance configured to connect to the RIS Live API.
// The logger is used for structured logging of connection events and errors.
// The metrics tracker is used to record message counts, bytes received, and
// connection status.
func New(logger *logger.Logger, metrics *metrics.Tracker) *Streamer {
return &Streamer{
logger: logger,
url: risLiveURL,
idleTimeout: streamIdleTimeout,
minBackoff: minBackoffDelay,
client: &http.Client{
Timeout: 0, // No timeout for streaming
Transport: &http.Transport{
// Disable automatic gzip decompression so we can measure wire bytes
DisableCompression: true,
},
},
handlers: make([]*handlerInfo, 0),
metrics: metrics,
//nolint:gosec // Non-cryptographic randomness is fine for backpressure
random: rand.New(rand.NewSource(time.Now().UnixNano())),
bgpPeers: make(map[string]bool),
}
}
// RegisterHandler adds a callback for message processing
// RegisterHandler adds a MessageHandler to receive parsed RIS messages.
// Each handler gets its own dedicated queue and worker goroutine for processing.
// If the streamer is already running, the handler's worker is started immediately.
func (s *Streamer) RegisterHandler(handler MessageHandler) {
s.mu.Lock()
defer s.mu.Unlock()
@@ -95,6 +174,10 @@ func (s *Streamer) RegisterHandler(handler MessageHandler) {
info := &handlerInfo{
handler: handler,
queue: make(chan *ristypes.RISMessage, handler.QueueCapacity()),
metrics: handlerMetrics{
// Initialize to max so first value sets the floor
minTime: time.Duration(math.MaxInt64),
},
}
s.handlers = append(s.handlers, info)
@@ -105,23 +188,30 @@ func (s *Streamer) RegisterHandler(handler MessageHandler) {
}
}
// RegisterRawHandler sets a callback for raw message lines
// RegisterRawHandler sets a callback to receive raw JSON lines from the stream
// before they are parsed. Only one raw handler can be registered at a time;
// subsequent calls will replace the previous handler.
func (s *Streamer) RegisterRawHandler(handler RawMessageHandler) {
s.mu.Lock()
defer s.mu.Unlock()
s.rawHandler = handler
}
// Start begins streaming in a goroutine
func (s *Streamer) Start() error {
// Start begins streaming BGP updates from the RIS Live API in a background goroutine.
// It starts worker goroutines for each registered handler and manages automatic
// reconnection with exponential backoff on connection failures.
// Returns an error if the streamer is already running. It streams until Stop:
// the end of ctx does not stop it.
func (s *Streamer) Start(ctx context.Context) error {
s.mu.Lock()
defer s.mu.Unlock()
if s.running {
return fmt.Errorf("streamer already running")
return errAlreadyRunning
}
ctx, cancel := context.WithCancel(context.Background())
ctx, cancel := context.WithCancel(context.WithoutCancel(ctx))
s.cancel = cancel
s.running = true
@@ -131,9 +221,7 @@ func (s *Streamer) Start() error {
}
go func() {
if err := s.stream(ctx); err != nil {
s.logger.Error("Streaming error", "error", err)
}
s.streamWithReconnect(ctx)
s.mu.Lock()
s.running = false
s.mu.Unlock()
@@ -142,47 +230,32 @@ func (s *Streamer) Start() error {
return nil
}
// Stop halts the streaming
// Stop halts the streaming connection and shuts down all handler workers.
// It cancels the streaming context, closes all handler queues, and updates
// the connection status in metrics. This method is safe to call multiple times.
func (s *Streamer) Stop() {
s.mu.Lock()
if s.cancel != nil {
s.cancel()
if s.cancel == nil {
// Not started, or already stopped: closing the queues again would panic.
s.mu.Unlock()
return
}
s.cancel()
s.cancel = nil
// Close all handler queues to signal workers to stop
for _, info := range s.handlers {
close(info.queue)
}
s.running = false
s.mu.Unlock()
s.metrics.SetConnected(false)
}
// runHandlerWorker processes messages for a specific handler
func (s *Streamer) runHandlerWorker(info *handlerInfo) {
for msg := range info.queue {
start := time.Now()
info.handler.HandleMessage(msg)
elapsed := time.Since(start)
// Update metrics
info.metrics.mu.Lock()
info.metrics.processedCount++
info.metrics.totalTime += elapsed
// Update min time
if info.metrics.minTime == 0 || elapsed < info.metrics.minTime {
info.metrics.minTime = elapsed
}
// Update max time
if elapsed > info.metrics.maxTime {
info.metrics.maxTime = elapsed
}
info.metrics.mu.Unlock()
}
}
// IsRunning returns whether the streamer is currently active
// IsRunning reports whether the streamer is currently connected and processing
// messages. This is safe to call concurrently from multiple goroutines.
func (s *Streamer) IsRunning() bool {
s.mu.RLock()
defer s.mu.RUnlock()
@@ -190,16 +263,26 @@ func (s *Streamer) IsRunning() bool {
return s.running
}
// GetMetrics returns current streaming metrics
// GetMetrics returns the current streaming metrics including message counts,
// bytes received, and throughput rates. The returned struct is a snapshot
// of the current state and is safe to use without synchronization.
func (s *Streamer) GetMetrics() metrics.StreamMetrics {
return s.metrics.GetStreamMetrics()
}
// HandlerStats represents metrics for a single handler
// GetMetricsTracker returns the underlying metrics.Tracker instance for direct access
// to metrics recording and retrieval functionality.
func (s *Streamer) GetMetricsTracker() *metrics.Tracker {
return s.metrics
}
// HandlerStats contains performance metrics for a single message handler.
// It includes queue utilization, message counts, and processing time statistics.
type HandlerStats struct {
Name string
QueueLength int
QueueCapacity int
QueueHighWaterMark int
ProcessedCount uint64
DroppedCount uint64
AvgProcessTime time.Duration
@@ -207,7 +290,9 @@ type HandlerStats struct {
MaxProcessTime time.Duration
}
// GetHandlerStats returns current handler statistics
// GetHandlerStats returns a snapshot of performance statistics for all registered
// handlers. The returned slice contains one HandlerStats entry per handler with
// current queue depth, processed/dropped counts, and processing time statistics.
func (s *Streamer) GetHandlerStats() []HandlerStats {
s.mu.RLock()
defer s.mu.RUnlock()
@@ -221,6 +306,7 @@ func (s *Streamer) GetHandlerStats() []HandlerStats {
Name: fmt.Sprintf("%T", info.handler),
QueueLength: len(info.queue),
QueueCapacity: cap(info.queue),
QueueHighWaterMark: info.metrics.queueHighWaterMark,
ProcessedCount: info.metrics.processedCount,
DroppedCount: info.metrics.droppedCount,
MinProcessTime: info.metrics.minTime,
@@ -230,12 +316,15 @@ func (s *Streamer) GetHandlerStats() []HandlerStats {
// Calculate average time
if info.metrics.processedCount > 0 {
processedCount := info.metrics.processedCount
const maxInt64 = 1<<63 - 1
if processedCount > maxInt64 {
processedCount = maxInt64
}
//nolint:gosec // processedCount is explicitly bounded above
hs.AvgProcessTime = info.metrics.totalTime / time.Duration(processedCount)
hs.AvgProcessTime = info.metrics.totalTime /
time.Duration(processedCount)
}
info.metrics.mu.Unlock()
@@ -246,9 +335,38 @@ func (s *Streamer) GetHandlerStats() []HandlerStats {
return stats
}
// GetDroppedMessages returns the total number of dropped messages
// GetDroppedMessages returns the total number of messages dropped across all
// handlers due to queue overflow or backpressure. This counter is monotonically
// increasing and is safe to call concurrently.
func (s *Streamer) GetDroppedMessages() uint64 {
return atomic.LoadUint64(&s.totalDropped)
return s.totalDropped.Load()
}
// runHandlerWorker processes messages for a specific handler
func (s *Streamer) runHandlerWorker(info *handlerInfo) {
for msg := range info.queue {
start := time.Now()
info.handler.HandleMessage(msg)
elapsed := time.Since(start)
// Update metrics
info.metrics.mu.Lock()
info.metrics.processedCount++
info.metrics.totalTime += elapsed
// Update min time
if elapsed < info.metrics.minTime {
info.metrics.minTime = elapsed
}
// Update max time
if elapsed > info.metrics.maxTime {
info.metrics.maxTime = elapsed
}
info.metrics.mu.Unlock()
}
}
// logMetrics logs the current streaming statistics
@@ -257,7 +375,8 @@ func (s *Streamer) logMetrics() {
uptime := time.Since(metrics.ConnectedSince)
const bitsPerMegabit = 1000000
totalDropped := atomic.LoadUint64(&s.totalDropped)
totalDropped := s.totalDropped.Load()
s.logger.Info(
"Stream statistics",
@@ -265,32 +384,37 @@ func (s *Streamer) logMetrics() {
uptime,
"total_messages",
metrics.TotalMessages,
"total_bytes",
"wire_bytes",
metrics.TotalWireBytes,
"wire_mb",
fmt.Sprintf("%.2f", float64(metrics.TotalWireBytes)/bytesPerMB),
"wire_mbps",
fmt.Sprintf("%.2f", metrics.WireBitsPerSec/bitsPerMegabit),
"decompressed_bytes",
metrics.TotalBytes,
"total_mb",
"decompressed_mb",
fmt.Sprintf("%.2f", float64(metrics.TotalBytes)/bytesPerMB),
"messages_per_sec",
fmt.Sprintf("%.2f", metrics.MessagesPerSec),
"bits_per_sec",
fmt.Sprintf("%.0f", metrics.BitsPerSec),
"mbps",
fmt.Sprintf("%.2f", metrics.BitsPerSec/bitsPerMegabit),
"total_dropped",
totalDropped,
)
// Log per-handler statistics
s.mu.RLock()
for i, info := range s.handlers {
info.metrics.mu.Lock()
if info.metrics.processedCount > 0 {
// Safe conversion: processedCount is bounded by maxInt64
processedCount := info.metrics.processedCount
const maxInt64 = 1<<63 - 1
if processedCount > maxInt64 {
processedCount = maxInt64
}
//nolint:gosec // processedCount is explicitly bounded above
avgTime := info.metrics.totalTime / time.Duration(processedCount)
s.logger.Info(
"Handler statistics",
@@ -307,6 +431,7 @@ func (s *Streamer) logMetrics() {
}
info.metrics.mu.Unlock()
}
s.mu.RUnlock()
}
@@ -315,27 +440,127 @@ func (s *Streamer) updateMetrics(messageBytes int) {
s.metrics.RecordMessage(int64(messageBytes))
}
// streamWithReconnect handles streaming with automatic reconnection and
// exponential backoff
func (s *Streamer) streamWithReconnect(ctx context.Context) {
backoffDelay := s.minBackoff
consecutiveFailures := 0
for {
select {
case <-ctx.Done():
s.logger.Info("Stream context cancelled, stopping reconnection attempts")
return
default:
}
// Attempt to stream
startTime := time.Now()
err := s.stream(ctx)
streamDuration := time.Since(startTime)
// Only Stop ends the streamer. A stream that ends any other way,
// with or without an error, is a disconnect and is reconnected.
if ctx.Err() != nil {
return
}
// Log the error
s.logger.Error("Stream disconnected",
"error", err,
"consecutive_failures", consecutiveFailures+1,
"stream_duration", streamDuration)
s.metrics.SetConnected(false)
// A stream that lasted more than 30 seconds resets the backoff. That
// includes a stream dropped after streamIdleTimeout with no line, so
// a silent feed is retried after the first backoff delay every time.
if streamDuration > 30*time.Second {
s.logger.Info("Resetting backoff delay due to successful connection",
"stream_duration", streamDuration)
backoffDelay = s.minBackoff
consecutiveFailures = 0
} else {
// Increment consecutive failures
consecutiveFailures++
}
// Wait with exponential backoff
s.logger.Info("Waiting before reconnection attempt",
"delay_seconds", backoffDelay.Seconds(),
"consecutive_failures", consecutiveFailures)
select {
case <-ctx.Done():
return
case <-time.After(backoffDelay):
// Double the backoff delay for next time, up to max
backoffDelay *= 2
if backoffDelay > maxBackoffDelay {
backoffDelay = maxBackoffDelay
}
}
}
}
func (s *Streamer) stream(ctx context.Context) error {
req, err := http.NewRequestWithContext(ctx, "GET", risLiveURL, nil)
// connCtx is scoped to this single connection: cancelling it when stream
// returns stops the ticker goroutines below, so a reconnect does not leak
// them. Without this they would live until the streamer's lifetime context
// is cancelled, leaking two per reconnect.
connCtx, connCancel := context.WithCancel(ctx)
defer connCancel()
// The idle timer drops the connection, by cancelling connCtx, when no
// line arrives for s.idleTimeout. Each line read starts it over. It
// also covers the wait for the server to answer.
idleTimer := time.AfterFunc(s.idleTimeout, connCancel)
defer idleTimer.Stop()
req, err := http.NewRequestWithContext(connCtx, http.MethodGet, s.url, nil)
if err != nil {
return fmt.Errorf("failed to create request: %w", err)
}
// Explicitly request gzip compression
req.Header.Set("Accept-Encoding", "gzip")
resp, err := s.client.Do(req)
if err != nil {
return fmt.Errorf("failed to connect to RIS Live: %w", err)
}
defer func() {
if err := resp.Body.Close(); err != nil {
err := resp.Body.Close()
if err != nil {
s.logger.Error("Failed to close response body", "error", err)
}
}()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("unexpected status code: %d", resp.StatusCode)
return fmt.Errorf("%w: %d", errUnexpectedStatus, resp.StatusCode)
}
// Wrap body with counting reader to track actual wire bytes
wireCounter := &countingReader{reader: resp.Body}
// Check if response is gzip-compressed and decompress if needed
var reader io.Reader = wireCounter
if resp.Header.Get("Content-Encoding") == "gzip" {
gzReader, err := gzip.NewReader(wireCounter)
if err != nil {
return fmt.Errorf("failed to create gzip reader: %w", err)
}
defer func() { _ = gzReader.Close() }()
reader = gzReader
s.logger.Info("Connected to RIS Live stream", "compression", "gzip")
} else {
s.logger.Info("Connected to RIS Live stream", "compression", "none")
}
s.logger.Info("Connected to RIS Live stream")
s.metrics.SetConnected(true)
// Start metrics logging goroutine
@@ -347,15 +572,56 @@ func (s *Streamer) stream(ctx context.Context) error {
select {
case <-metricsTicker.C:
s.logMetrics()
case <-ctx.Done():
case <-connCtx.Done():
return
}
}
}()
scanner := bufio.NewScanner(resp.Body)
// Wire byte update ticker - update metrics with actual wire bytes periodically
go s.recordWireBytes(connCtx, wireCounter)
return s.readStream(ctx, connCtx, reader, idleTimer)
}
// recordWireBytes adds the bytes wireCounter has counted to the metrics once a
// second until ctx is cancelled.
func (s *Streamer) recordWireBytes(
ctx context.Context, wireCounter *countingReader,
) {
wireUpdateTicker := time.NewTicker(time.Second)
defer wireUpdateTicker.Stop()
var lastWireBytes int64
for {
select {
case <-wireUpdateTicker.C:
currentBytes := wireCounter.Count()
delta := currentBytes - lastWireBytes
if delta > 0 {
s.metrics.RecordWireBytes(delta)
lastWireBytes = currentBytes
}
case <-ctx.Done():
return
}
}
}
// readStream hands each line of the stream to processLine until the stream
// ends, a line fails, or ctx is cancelled. Each line restarts idleTimer, which
// cancels connCtx when it fires.
func (s *Streamer) readStream(
ctx, connCtx context.Context, reader io.Reader, idleTimer *time.Timer,
) error {
scanner := bufio.NewScanner(reader)
for scanner.Scan() {
idleTimer.Reset(s.idleTimeout)
select {
case <-ctx.Done():
s.logger.Info("Stream stopped by context")
@@ -369,7 +635,33 @@ func (s *Streamer) stream(ctx context.Context) error {
continue
}
// Update metrics with message size
err := s.processLine(ctx, line)
if err != nil {
return err
}
}
if ctx.Err() != nil {
return ctx.Err()
}
if connCtx.Err() != nil {
return fmt.Errorf("%w for %s", errStreamIdle, s.idleTimeout)
}
err := scanner.Err()
if err != nil {
return fmt.Errorf("scanner error: %w", err)
}
return errStreamEnded
}
// processLine passes one line of the stream to the raw handler, parses it,
// and dispatches an UPDATE message to the handlers. An error ends the
// connection.
func (s *Streamer) processLine(ctx context.Context, line []byte) error {
// Update metrics with decompressed message size
s.updateMetrics(len(line))
// Call raw handler if registered
@@ -384,11 +676,16 @@ func (s *Streamer) stream(ctx context.Context) error {
// Parse the message first
var wrapper ristypes.RISLiveMessage
if err := json.Unmarshal(line, &wrapper); err != nil {
// Output the raw line and panic on parse failure
fmt.Fprintf(os.Stderr, "Failed to parse JSON: %v\n", err)
fmt.Fprintf(os.Stderr, "Raw line: %s\n", string(line))
panic(fmt.Sprintf("JSON parse error: %v", err))
err := json.Unmarshal(line, &wrapper)
if err != nil {
// Log the error and return to trigger reconnection
s.logger.Error("Failed to parse JSON",
"error", err,
"line", string(line),
"line_length", len(line))
return fmt.Errorf("JSON parse error: %w", err)
}
// Check if it's a ris_message wrapper
@@ -398,7 +695,7 @@ func (s *Streamer) stream(ctx context.Context) error {
"line", string(line),
)
continue
return nil
}
// Get the actual message
@@ -414,64 +711,125 @@ func (s *Streamer) stream(ctx context.Context) error {
// Will be dispatched to handlers
case "RIS_PEER_STATE":
// RIS peer state messages - silently ignore
continue
return nil
case "KEEPALIVE":
// BGP keepalive messages - silently process
continue
return nil
case "OPEN":
// BGP open messages
// BGP open messages - track peer as active
s.bgpPeersMu.Lock()
s.bgpPeers[msg.Peer] = true
peerCount := len(s.bgpPeers)
s.bgpPeersMu.Unlock()
s.metrics.SetBGPPeerCount(peerCount)
s.logger.Info("BGP session opened",
"peer", msg.Peer,
"peer_asn", msg.PeerASN,
"total_peers", peerCount,
)
continue
return nil
case "NOTIFICATION":
// BGP notification messages (errors)
// BGP notification messages (session closed)
s.bgpPeersMu.Lock()
delete(s.bgpPeers, msg.Peer)
peerCount := len(s.bgpPeers)
s.bgpPeersMu.Unlock()
s.metrics.SetBGPPeerCount(peerCount)
s.logger.Warn("BGP notification",
"peer", msg.Peer,
"peer_asn", msg.PeerASN,
"total_peers", peerCount,
)
continue
return nil
case "STATE":
// Peer state changes - silently ignore
continue
return nil
default:
fmt.Fprintf(
os.Stderr,
"UNKNOWN MESSAGE TYPE: %s\nRAW MESSAGE: %s\n",
msg.Type,
string(line),
)
panic(
fmt.Sprintf(
"Unknown RIS message type: %s",
msg.Type,
),
s.logger.Warn("Unknown message type, skipping",
"type", msg.Type,
)
return nil
}
// Dispatch to interested handlers
return s.dispatchMessage(ctx, &msg)
}
// dispatchMessage queues msg for each handler that wants it, dropping it for
// a handler whose queue is full or under backpressure.
func (s *Streamer) dispatchMessage(
ctx context.Context, msg *ristypes.RISMessage,
) error {
// Stop cancels ctx and closes the queues under the write lock, so if ctx
// is cancelled here, under the read lock, the queues are closed and must
// not be sent to.
s.mu.RLock()
if ctx.Err() != nil {
s.mu.RUnlock()
return ctx.Err()
}
for _, info := range s.handlers {
if info.handler.WantsMessage(msg.Type) {
if !info.handler.WantsMessage(msg.Type) {
continue
}
// Check if we should drop due to backpressure
if s.shouldDropForBackpressure(info) {
atomic.AddUint64(&info.metrics.droppedCount, 1)
s.totalDropped.Add(1)
continue
}
// Try to queue the message
select {
case info.queue <- &msg:
case info.queue <- msg:
// Message queued successfully
// Update high water mark if needed
queueLen := len(info.queue)
info.metrics.mu.Lock()
if queueLen > info.metrics.queueHighWaterMark {
info.metrics.queueHighWaterMark = queueLen
}
info.metrics.mu.Unlock()
default:
// Queue is full, drop the message
atomic.AddUint64(&info.metrics.droppedCount, 1)
atomic.AddUint64(&s.totalDropped, 1)
s.totalDropped.Add(1)
}
}
}
s.mu.RUnlock()
}
if err := scanner.Err(); err != nil {
return fmt.Errorf("scanner error: %w", err)
}
s.mu.RUnlock()
return nil
}
// shouldDropForBackpressure determines if a message should be dropped based on
// queue utilization
func (s *Streamer) shouldDropForBackpressure(info *handlerInfo) bool {
// Calculate queue utilization
queueLen := len(info.queue)
queueCap := cap(info.queue)
utilization := float64(queueLen) / float64(queueCap)
// No drops below threshold
if utilization < backpressureThreshold {
return false
}
// Calculate drop probability (0.0 at threshold, 1.0 at 100% full)
dropProbability := (utilization - backpressureThreshold) * backpressureSlope
if dropProbability > 1.0 {
dropProbability = 1.0
}
// Random drop based on probability
return s.random.Float64() < dropProbability
}
internal/streamer/streamer_internal_test.go
+332
View File
@@ -0,0 +1,332 @@
package streamer
import (
"context"
"errors"
"io"
"net/http"
"net/http/httptest"
"runtime"
"sync/atomic"
"testing"
"time"
"git.eeqj.de/sneak/routewatch/internal/logger"
"git.eeqj.de/sneak/routewatch/internal/metrics"
"git.eeqj.de/sneak/routewatch/internal/ristypes"
)
func TestNewStreamer(t *testing.T) {
t.Parallel()
logger := logger.New()
metricsTracker := metrics.New()
s := New(logger, metricsTracker)
if s == nil {
t.Fatal("New() returned nil")
}
if s.logger != logger {
t.Error("logger not set correctly")
}
if s.client == nil {
t.Error("HTTP client not initialized")
}
if s.handlers == nil {
t.Error("handlers slice not initialized")
}
if s.metrics != metricsTracker {
t.Error("metrics tracker not set correctly")
}
if s.idleTimeout != streamIdleTimeout {
t.Errorf("idleTimeout = %s, want %s", s.idleTimeout, streamIdleTimeout)
}
if s.minBackoff != minBackoffDelay {
t.Errorf("minBackoff = %s, want %s", s.minBackoff, minBackoffDelay)
}
}
// TestStreamDoesNotLeakTickersAcrossReconnects drives many short-lived
// connections (each stream call is one reconnect cycle) and asserts the
// goroutine count returns to its starting value. Each connection starts two
// ticker goroutines; before the fix they lived until the streamer's lifetime
// context was cancelled, so every reconnect leaked two.
//
//nolint:paralleltest // counts goroutines, so no other test may run alongside
func TestStreamDoesNotLeakTickersAcrossReconnects(t *testing.T) {
// The handler returns immediately, so the response body is empty and each
// stream call ends at once, standing in for a dropped connection.
srv := httptest.NewServer(http.HandlerFunc(
func(_ http.ResponseWriter, _ *http.Request) {}))
defer srv.Close()
s := New(logger.New(), metrics.New())
s.url = srv.URL
// One warm-up connection so any persistent HTTP transport goroutine exists
// before we take the baseline.
err := s.stream(context.Background())
if !errors.Is(err, errStreamEnded) {
t.Fatalf("warm-up stream returned %v, want %v", err, errStreamEnded)
}
s.client.CloseIdleConnections()
baseline := settledGoroutineCount()
const reconnects = 20
for range reconnects {
err := s.stream(context.Background())
if !errors.Is(err, errStreamEnded) {
t.Fatalf("stream returned %v, want %v", err, errStreamEnded)
}
}
s.client.CloseIdleConnections()
if !waitForGoroutines(baseline) {
t.Fatalf("goroutines did not return to baseline %d after %d reconnects, got %d",
baseline, reconnects, runtime.NumGoroutine())
}
}
// updateLine is one UPDATE message as RIS Live sends it.
const updateLine = `{"type":"ris_message","data":{"type":"UPDATE",` +
`"peer":"192.0.2.1","peer_asn":"64496","timestamp":1700000000}}` + "\n"
// updateHandler wants UPDATE messages. For each one it handles it sends on
// handled if there is room, and otherwise does nothing; a nil handled is
// never sent on.
type updateHandler struct {
handled chan struct{}
}
func (updateHandler) WantsMessage(messageType string) bool {
return messageType == "UPDATE"
}
func (updateHandler) QueueCapacity() int { return 10 }
func (h updateHandler) HandleMessage(*ristypes.RISMessage) {
select {
case h.handled <- struct{}{}:
default:
}
}
// flush sends what w has buffered to the client. The writers httptest servers
// hand to handlers can always do that.
func flush(w http.ResponseWriter) {
flusher, ok := w.(http.Flusher)
if ok {
flusher.Flush()
}
}
// TestStopBeforeMessageReachesQueues stops the streamer after the read loop
// has checked for cancellation but before it hands the message to the handler
// queues. That is the gap a stop from another goroutine can land in, and it
// used to end in "send on closed channel". The raw handler runs in that gap on
// the read loop itself, so calling Stop from it hits the gap every time.
func TestStopBeforeMessageReachesQueues(t *testing.T) {
t.Parallel()
srv := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
_, _ = io.WriteString(w, updateLine)
}))
defer srv.Close()
s := New(logger.New(), metrics.New())
s.url = srv.URL
s.RegisterHandler(updateHandler{})
s.RegisterRawHandler(func(string) { s.Stop() })
// Start would run the stream in the background, where the test cannot
// wait for it. Setting cancel as Start does lets Stop cancel the stream
// run here instead.
ctx, cancel := context.WithCancel(context.Background())
s.cancel = cancel
err := s.stream(ctx)
if !errors.Is(err, context.Canceled) {
t.Fatalf("stream returned %v, want %v", err, context.Canceled)
}
// A second Stop must not close the queues again.
s.Stop()
}
// TestReconnectWhenStreamEnds has the server end the first response
// normally, as when RIS Live restarts. The streamer must treat that as a
// disconnect and connect again.
func TestReconnectWhenStreamEnds(t *testing.T) {
t.Parallel()
endResponse := func(http.ResponseWriter, *http.Request) {}
checkReconnect(t, endResponse, streamIdleTimeout)
}
// TestReconnectWhenStreamGoesIdle has the server hold the first response
// open without sending anything. The streamer must drop it once the idle
// timeout passes and connect again.
func TestReconnectWhenStreamGoesIdle(t *testing.T) {
t.Parallel()
const idleTimeout = 50 * time.Millisecond
holdResponse := func(w http.ResponseWriter, r *http.Request) {
flush(w)
<-r.Context().Done()
}
checkReconnect(t, holdResponse, idleTimeout)
}
// TestReconnectWhenServerNeverAnswers has the server accept the first
// request and never answer it, not even with headers. The streamer must give
// up on it once the idle timeout passes and connect again.
func TestReconnectWhenServerNeverAnswers(t *testing.T) {
t.Parallel()
const idleTimeout = 50 * time.Millisecond
neverAnswer := func(_ http.ResponseWriter, r *http.Request) {
<-r.Context().Done()
}
checkReconnect(t, neverAnswer, idleTimeout)
}
// TestStreamNotDroppedWhileLinesArrive has the server send lines well within
// the idle timeout for several times the timeout, then end the response. Each
// line must start the idle timeout over, so the stream lasts until the server
// ends it instead of being dropped as idle.
func TestStreamNotDroppedWhileLinesArrive(t *testing.T) {
t.Parallel()
const (
idleTimeout = 100 * time.Millisecond
lineInterval = idleTimeout / 10
sendFor = 5 * idleTimeout
)
srv := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
deadline := time.Now().Add(sendFor)
for time.Now().Before(deadline) {
_, _ = io.WriteString(w, updateLine)
flush(w)
time.Sleep(lineInterval)
}
}))
defer srv.Close()
s := New(logger.New(), metrics.New())
s.url = srv.URL
s.idleTimeout = idleTimeout
err := s.stream(context.Background())
if !errors.Is(err, errStreamEnded) {
t.Fatalf("stream returned %v, want %v", err, errStreamEnded)
}
}
// checkReconnect runs a streamer against a fake RIS Live server. The first
// response is served by firstResponse; every later one carries one UPDATE
// line and then stays open. The streamer must come back for a second
// response, must not be marked connected when it does, and must handle the
// UPDATE in it.
func checkReconnect(
t *testing.T, firstResponse http.HandlerFunc, idleTimeout time.Duration,
) {
t.Helper()
const maxWait = 5 * time.Second
tracker := metrics.New()
connectedAtReconnect := make(chan bool, 1)
var requests atomic.Int32
srv := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
if requests.Add(1) == 1 {
firstResponse(w, r)
return
}
select {
case connectedAtReconnect <- tracker.IsConnected():
default:
}
_, _ = io.WriteString(w, updateLine)
flush(w)
<-r.Context().Done()
}))
defer srv.Close()
s := New(logger.New(), tracker)
s.url = srv.URL
s.idleTimeout = idleTimeout
s.minBackoff = time.Millisecond
handler := updateHandler{handled: make(chan struct{}, 1)}
s.RegisterHandler(handler)
err := s.Start(t.Context())
if err != nil {
t.Fatalf("Start returned error: %v", err)
}
defer s.Stop()
select {
case <-handler.handled:
case <-time.After(maxWait):
t.Fatalf("no message from a second response handled within %s", maxWait)
}
if <-connectedAtReconnect {
t.Error("streamer was still marked connected when it came back " +
"for a second response")
}
}
// settledGoroutineCount lets transient goroutines finish, then reports the
// current count.
func settledGoroutineCount() int {
prev := runtime.NumGoroutine()
for range 20 {
time.Sleep(10 * time.Millisecond)
cur := runtime.NumGoroutine()
if cur == prev {
return cur
}
prev = cur
}
return prev
}
// waitForGoroutines waits until the goroutine count drops to target or below.
func waitForGoroutines(target int) bool {
for range 100 {
if runtime.NumGoroutine() <= target {
return true
}
time.Sleep(10 * time.Millisecond)
}
return false
}
internal/streamer/streamer_test.go
-34
View File
@@ -1,34 +0,0 @@
package streamer
import (
"testing"
"git.eeqj.de/sneak/routewatch/internal/metrics"
"log/slog"
)
func TestNewStreamer(t *testing.T) {
logger := slog.Default()
metricsTracker := metrics.New()
s := New(logger, metricsTracker)
if s == nil {
t.Fatal("New() returned nil")
}
if s.logger != logger {
t.Error("logger not set correctly")
}
if s.client == nil {
t.Error("HTTP client not initialized")
}
if s.handlers == nil {
t.Error("handlers slice not initialized")
}
if s.metrics != metricsTracker {
t.Error("metrics tracker not set correctly")
}
}
internal/templates/as_detail.html
+346
View File
@@ -0,0 +1,346 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>AS{{.ASN.ASN}} - {{.ASN.Handle}} - RouteWatch</title>
<style>
* {
box-sizing: border-box;
}
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
margin: 0;
padding: 0;
background: #f5f5f5;
color: #333;
}
/* Navbar styles */
.navbar {
background: #2c3e50;
padding: 0 20px;
display: flex;
align-items: center;
justify-content: space-between;
height: 56px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
}
.navbar-brand {
display: flex;
align-items: center;
gap: 8px;
font-size: 18px;
font-weight: 600;
}
.navbar-brand a {
color: white;
text-decoration: none;
}
.navbar-brand a:hover {
color: #ecf0f1;
}
.navbar-brand .by {
font-weight: normal;
color: #95a5a6;
font-size: 14px;
}
.navbar-brand .author {
color: #3498db;
font-weight: normal;
}
.navbar-brand .author:hover {
text-decoration: underline;
}
.navbar-links {
display: flex;
gap: 20px;
}
.navbar-links a {
color: #ecf0f1;
text-decoration: none;
font-size: 14px;
padding: 8px 12px;
border-radius: 4px;
transition: background-color 0.2s;
}
.navbar-links a:hover {
background: rgba(255,255,255,0.1);
}
.navbar-links a.active {
background: rgba(255,255,255,0.15);
}
/* Main content */
.main-content {
padding: 20px;
}
.container {
max-width: 1200px;
margin: 0 auto;
background: white;
padding: 30px;
border-radius: 8px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
}
h1 {
margin: 0 0 10px 0;
color: #2c3e50;
}
.subtitle {
color: #7f8c8d;
margin-bottom: 30px;
}
.info-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));
gap: 20px;
margin-bottom: 30px;
}
.info-card {
background: #f8f9fa;
padding: 20px;
border-radius: 6px;
border-left: 4px solid #3498db;
}
.info-label {
font-size: 14px;
color: #7f8c8d;
margin-bottom: 5px;
}
.info-value {
font-size: 24px;
font-weight: bold;
color: #2c3e50;
}
.prefix-section {
margin-top: 30px;
}
.prefix-header {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: 20px;
}
.prefix-header h2 {
margin: 0;
color: #2c3e50;
}
.prefix-count {
background: #e74c3c;
color: white;
padding: 5px 12px;
border-radius: 20px;
font-size: 14px;
font-weight: bold;
}
.prefix-table {
width: 100%;
border-collapse: collapse;
background: white;
border: 1px solid #e0e0e0;
border-radius: 6px;
overflow: hidden;
}
.prefix-table th {
background: #34495e;
color: white;
padding: 12px;
text-align: left;
font-weight: 600;
}
.prefix-table td {
padding: 12px;
border-bottom: 1px solid #e0e0e0;
}
.prefix-table tr:hover {
background: #f8f9fa;
}
.prefix-table tr:last-child td {
border-bottom: none;
}
.prefix-link {
color: #3498db;
text-decoration: none;
font-family: monospace;
}
.prefix-link:hover {
text-decoration: underline;
}
.age {
color: #7f8c8d;
font-size: 14px;
}
.nav-link {
display: inline-block;
margin-bottom: 20px;
color: #3498db;
text-decoration: none;
}
.nav-link:hover {
text-decoration: underline;
}
.empty-state {
text-align: center;
padding: 40px;
color: #7f8c8d;
}
@media (max-width: 768px) {
.container {
padding: 20px;
}
.info-grid {
grid-template-columns: 1fr;
}
}
</style>
</head>
<body>
<nav class="navbar">
<div class="navbar-brand">
<a href="/">routewatch</a>
<span class="by">by</span>
<a href="https://sneak.berlin" class="author">@sneak</a>
</div>
<div class="navbar-links">
<a href="/">Home</a>
<a href="/status">Status</a>
</div>
</nav>
<main class="main-content">
<div class="container">
<h1>AS{{.ASN.ASN}}{{if .ASN.Handle}} - {{.ASN.Handle}}{{end}}</h1>
{{if .ASN.Description}}
<p class="subtitle">{{.ASN.Description}}</p>
{{end}}
<div class="info-grid">
<div class="info-card">
<div class="info-label">Total Prefixes</div>
<div class="info-value">{{.TotalCount}}</div>
</div>
<div class="info-card">
<div class="info-label">IPv4 Prefixes</div>
<div class="info-value">{{.IPv4Count}}</div>
</div>
<div class="info-card">
<div class="info-label">IPv6 Prefixes</div>
<div class="info-value">{{.IPv6Count}}</div>
</div>
<div class="info-card">
<div class="info-label">Peer ASNs</div>
<div class="info-value">{{.PeerCount}}</div>
</div>
<div class="info-card">
<div class="info-label">First Seen</div>
<div class="info-value">{{.ASN.FirstSeen.Format "2006-01-02"}}</div>
</div>
</div>
{{if .IPv4Prefixes}}
<div class="prefix-section">
<div class="prefix-header">
<h2>IPv4 Prefixes</h2>
<span class="prefix-count">{{.IPv4Count}}</span>
</div>
<table class="prefix-table">
<thead>
<tr>
<th>Prefix</th>
<th>Mask Length</th>
<th>Last Updated</th>
<th>Age</th>
</tr>
</thead>
<tbody>
{{range .IPv4Prefixes}}
<tr>
<td><a href="{{.Prefix | prefixURL}}" class="prefix-link">{{.Prefix}}</a></td>
<td>/{{.MaskLength}}</td>
<td>{{.LastUpdated.Format "2006-01-02 15:04:05"}}</td>
<td class="age">{{.LastUpdated | timeSince}}</td>
</tr>
{{end}}
</tbody>
</table>
</div>
{{end}}
{{if .IPv6Prefixes}}
<div class="prefix-section">
<div class="prefix-header">
<h2>IPv6 Prefixes</h2>
<span class="prefix-count">{{.IPv6Count}}</span>
</div>
<table class="prefix-table">
<thead>
<tr>
<th>Prefix</th>
<th>Mask Length</th>
<th>Last Updated</th>
<th>Age</th>
</tr>
</thead>
<tbody>
{{range .IPv6Prefixes}}
<tr>
<td><a href="{{.Prefix | prefixURL}}" class="prefix-link">{{.Prefix}}</a></td>
<td>/{{.MaskLength}}</td>
<td>{{.LastUpdated.Format "2006-01-02 15:04:05"}}</td>
<td class="age">{{.LastUpdated | timeSince}}</td>
</tr>
{{end}}
</tbody>
</table>
</div>
{{end}}
{{if eq .TotalCount 0}}
<div class="empty-state">
<p>No prefixes announced by this AS</p>
</div>
{{end}}
{{if .Peers}}
<div class="prefix-section">
<div class="prefix-header">
<h2>Peer ASNs</h2>
<span class="prefix-count">{{.PeerCount}}</span>
</div>
<table class="prefix-table">
<thead>
<tr>
<th>ASN</th>
<th>Handle</th>
<th>Description</th>
<th>First Seen</th>
<th>Last Seen</th>
</tr>
</thead>
<tbody>
{{range .Peers}}
<tr>
<td><a href="/as/{{.ASN}}" class="prefix-link">AS{{.ASN}}</a></td>
<td>{{if .Handle}}{{.Handle}}{{else}}-{{end}}</td>
<td>{{if .Description}}{{.Description}}{{else}}-{{end}}</td>
<td>{{.FirstSeen.Format "2006-01-02"}}</td>
<td>{{.LastSeen.Format "2006-01-02"}}</td>
</tr>
{{end}}
</tbody>
</table>
</div>
{{else}}
<div class="prefix-section">
<h2>Peer ASNs</h2>
<div class="empty-state">
<p>No peering relationships found for this AS</p>
</div>
</div>
{{end}}
</div>
</main>
</body>
</html>
internal/templates/index.html
+447
View File
@@ -0,0 +1,447 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>RouteWatch - BGP Route Monitor</title>
<style>
* {
box-sizing: border-box;
}
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
margin: 0;
padding: 0;
background: #f5f5f5;
color: #333;
}
/* Navbar styles */
.navbar {
background: #2c3e50;
padding: 0 20px;
display: flex;
align-items: center;
justify-content: space-between;
height: 56px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
}
.navbar-brand {
display: flex;
align-items: center;
gap: 8px;
font-size: 18px;
font-weight: 600;
}
.navbar-brand a {
color: white;
text-decoration: none;
}
.navbar-brand a:hover {
color: #ecf0f1;
}
.navbar-brand .by {
font-weight: normal;
color: #95a5a6;
font-size: 14px;
}
.navbar-brand .author {
color: #3498db;
font-weight: normal;
}
.navbar-brand .author:hover {
text-decoration: underline;
}
.navbar-links {
display: flex;
gap: 20px;
}
.navbar-links a {
color: #ecf0f1;
text-decoration: none;
font-size: 14px;
padding: 8px 12px;
border-radius: 4px;
transition: background-color 0.2s;
}
.navbar-links a:hover {
background: rgba(255,255,255,0.1);
}
.navbar-links a.active {
background: rgba(255,255,255,0.15);
}
/* Main content */
.main-content {
max-width: 1200px;
margin: 0 auto;
padding: 30px 20px;
}
/* Stats overview */
.stats-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
gap: 20px;
margin-bottom: 40px;
}
.stat-card {
background: white;
padding: 24px;
border-radius: 8px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
text-align: center;
}
.stat-value {
font-size: 32px;
font-weight: bold;
color: #2c3e50;
margin-bottom: 8px;
}
.stat-label {
font-size: 14px;
color: #7f8c8d;
text-transform: uppercase;
letter-spacing: 0.5px;
}
.stat-card.connected .stat-value {
color: #27ae60;
}
.stat-card.disconnected .stat-value {
color: #e74c3c;
}
/* Search section */
.search-section {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(350px, 1fr));
gap: 24px;
margin-bottom: 40px;
}
.search-card {
background: white;
padding: 24px;
border-radius: 8px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
}
.search-card h2 {
margin: 0 0 16px 0;
font-size: 18px;
color: #2c3e50;
}
.search-input-group {
display: flex;
gap: 10px;
}
.search-input-group input {
flex: 1;
padding: 12px 16px;
border: 1px solid #ddd;
border-radius: 6px;
font-size: 14px;
outline: none;
transition: border-color 0.2s;
}
.search-input-group input:focus {
border-color: #3498db;
}
.search-input-group button {
padding: 12px 24px;
background: #3498db;
color: white;
border: none;
border-radius: 6px;
font-size: 14px;
cursor: pointer;
transition: background-color 0.2s;
}
.search-input-group button:hover {
background: #2980b9;
}
.search-input-group button:disabled {
background: #bdc3c7;
cursor: not-allowed;
}
.search-hint {
font-size: 12px;
color: #95a5a6;
margin-top: 8px;
}
/* IP Lookup result */
.ip-result {
margin-top: 16px;
display: none;
}
.ip-result.visible {
display: block;
}
.ip-result-header {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: 8px;
}
.ip-result-header h3 {
margin: 0;
font-size: 14px;
color: #2c3e50;
}
.ip-result-header button {
background: none;
border: none;
color: #e74c3c;
cursor: pointer;
font-size: 12px;
}
.ip-result pre {
background: #2c3e50;
color: #ecf0f1;
padding: 16px;
border-radius: 6px;
overflow-x: auto;
font-size: 12px;
line-height: 1.5;
margin: 0;
max-height: 400px;
overflow-y: auto;
}
.ip-result .error {
background: #fee;
color: #c00;
padding: 12px;
border-radius: 6px;
font-size: 14px;
}
.ip-result .loading {
color: #7f8c8d;
font-style: italic;
}
/* Footer */
.footer {
margin-top: 40px;
padding: 20px;
background: white;
border-radius: 8px;
box-shadow: 0 -2px 10px rgba(0,0,0,0.05);
text-align: center;
color: #7f8c8d;
font-size: 14px;
}
.footer a {
color: #3498db;
text-decoration: none;
}
.footer a:hover {
text-decoration: underline;
}
.footer .separator {
margin: 0 10px;
color: #ddd;
}
</style>
</head>
<body>
<nav class="navbar">
<div class="navbar-brand">
<a href="/">routewatch</a>
<span class="by">by</span>
<a href="https://sneak.berlin" class="author">@sneak</a>
</div>
<div class="navbar-links">
<a href="/" class="active">Home</a>
<a href="/status">Status</a>
</div>
</nav>
<main class="main-content">
<div class="stats-grid">
<div class="stat-card" id="status-card">
<div class="stat-value" id="stat-status">-</div>
<div class="stat-label">Status</div>
</div>
<div class="stat-card">
<div class="stat-value" id="stat-routes">-</div>
<div class="stat-label">Live Routes</div>
</div>
<div class="stat-card">
<div class="stat-value" id="stat-asns">-</div>
<div class="stat-label">Autonomous Systems</div>
</div>
<div class="stat-card">
<div class="stat-value" id="stat-prefixes">-</div>
<div class="stat-label">Prefixes</div>
</div>
<div class="stat-card">
<div class="stat-value" id="stat-peers">-</div>
<div class="stat-label">BGP Peers</div>
</div>
<div class="stat-card">
<div class="stat-value" id="stat-updates">-</div>
<div class="stat-label">Updates/sec</div>
</div>
</div>
<div class="search-section">
<div class="search-card">
<h2>AS Number Lookup</h2>
<form id="asn-form" class="search-input-group">
<input type="text" id="asn-input" placeholder="e.g., 15169 or AS15169" autocomplete="off">
<button type="submit">Lookup</button>
</form>
<p class="search-hint">Enter an AS number to view its announced prefixes and peers</p>
</div>
<div class="search-card">
<h2>AS Name Search</h2>
<form id="asname-form" class="search-input-group">
<input type="text" id="asname-input" placeholder="e.g., Google, Cloudflare" autocomplete="off">
<button type="submit">Search</button>
</form>
<p class="search-hint">Search for autonomous systems by organization name</p>
<div id="asname-results"></div>
</div>
<div class="search-card">
<h2>IP Address Lookup</h2>
<form id="ip-form" class="search-input-group">
<input type="text" id="ip-input" placeholder="e.g., 8.8.8.8 or 2001:4860:4860::8888" autocomplete="off">
<button type="submit">Lookup</button>
</form>
<p class="search-hint">Get routing information for any IP address</p>
<div id="ip-result" class="ip-result">
<div class="ip-result-header">
<h3>Result</h3>
<button type="button" id="ip-result-close">Clear</button>
</div>
<pre id="ip-result-content"></pre>
</div>
</div>
</div>
</main>
<footer class="footer">
<span><a href="{{appRepoURL}}">{{appName}}</a> by <a href="{{appAuthorURL}}">{{appAuthor}}</a></span>
<span class="separator">|</span>
<span>{{appLicense}}</span>
<span class="separator">|</span>
<span><a href="{{appGitCommitURL}}">{{appGitRevision}}</a></span>
</footer>
<script>
function formatNumber(num) {
if (num >= 1000000) {
return (num / 1000000).toFixed(1) + 'M';
} else if (num >= 1000) {
return (num / 1000).toFixed(1) + 'K';
}
return num.toLocaleString();
}
// Fetch and display stats
function updateStats() {
fetch('/api/v1/stats')
.then(response => response.json())
.then(response => {
if (response.status !== 'ok') return;
const data = response.data;
const statusCard = document.getElementById('status-card');
const statusEl = document.getElementById('stat-status');
statusEl.textContent = data.connected ? 'Connected' : 'Disconnected';
statusCard.className = 'stat-card ' + (data.connected ? 'connected' : 'disconnected');
document.getElementById('stat-routes').textContent = formatNumber(data.live_routes);
document.getElementById('stat-asns').textContent = formatNumber(data.asns);
document.getElementById('stat-prefixes').textContent = formatNumber(data.prefixes);
if (data.stream) {
document.getElementById('stat-peers').textContent = formatNumber(data.stream.bgp_peer_count);
}
const totalUpdates = data.ipv4_updates_per_sec + data.ipv6_updates_per_sec;
document.getElementById('stat-updates').textContent = totalUpdates.toFixed(1);
})
.catch(() => {
document.getElementById('stat-status').textContent = 'Error';
document.getElementById('status-card').className = 'stat-card disconnected';
});
}
// ASN lookup
document.getElementById('asn-form').addEventListener('submit', function(e) {
e.preventDefault();
let asn = document.getElementById('asn-input').value.trim();
// Remove 'AS' prefix if present
asn = asn.replace(/^AS/i, '');
if (asn && /^\d+$/.test(asn)) {
window.location.href = '/as/' + asn;
}
});
// AS name search
document.getElementById('asname-form').addEventListener('submit', function(e) {
e.preventDefault();
const query = document.getElementById('asname-input').value.trim();
if (!query) return;
const resultsDiv = document.getElementById('asname-results');
resultsDiv.innerHTML = '<p class="loading" style="color: #7f8c8d; margin-top: 12px;">Searching...</p>';
// Use a simple client-side search against the asinfo data
// For now, redirect to AS page if it looks like an ASN
if (/^\d+$/.test(query)) {
window.location.href = '/as/' + query;
return;
}
// Show a message that server-side search is coming
resultsDiv.innerHTML = '<p style="color: #7f8c8d; margin-top: 12px; font-size: 13px;">AS name search coming soon. For now, try an AS number.</p>';
});
// IP lookup
document.getElementById('ip-form').addEventListener('submit', function(e) {
e.preventDefault();
const ip = document.getElementById('ip-input').value.trim();
if (!ip) return;
const resultDiv = document.getElementById('ip-result');
const contentEl = document.getElementById('ip-result-content');
resultDiv.classList.add('visible');
contentEl.className = '';
contentEl.textContent = 'Loading...';
contentEl.classList.add('loading');
fetch('/ip/' + encodeURIComponent(ip))
.then(response => response.json())
.then(response => {
contentEl.classList.remove('loading');
if (response.status === 'error') {
contentEl.className = 'error';
contentEl.textContent = 'Error: ' + response.error.msg;
} else {
contentEl.className = '';
contentEl.textContent = JSON.stringify(response.data, null, 2);
}
})
.catch(error => {
contentEl.classList.remove('loading');
contentEl.className = 'error';
contentEl.textContent = 'Error: ' + error.message;
});
});
// Close IP result
document.getElementById('ip-result-close').addEventListener('click', function() {
document.getElementById('ip-result').classList.remove('visible');
document.getElementById('ip-input').value = '';
});
// Initial load and refresh stats every 5 seconds
updateStats();
setInterval(updateStats, 5000);
</script>
</body>
</html>
internal/templates/prefix_detail.html
+335
View File
@@ -0,0 +1,335 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>{{.Prefix}} - RouteWatch</title>
<style>
* {
box-sizing: border-box;
}
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
margin: 0;
padding: 0;
background: #f5f5f5;
color: #333;
}
/* Navbar styles */
.navbar {
background: #2c3e50;
padding: 0 20px;
display: flex;
align-items: center;
justify-content: space-between;
height: 56px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
}
.navbar-brand {
display: flex;
align-items: center;
gap: 8px;
font-size: 18px;
font-weight: 600;
}
.navbar-brand a {
color: white;
text-decoration: none;
}
.navbar-brand a:hover {
color: #ecf0f1;
}
.navbar-brand .by {
font-weight: normal;
color: #95a5a6;
font-size: 14px;
}
.navbar-brand .author {
color: #3498db;
font-weight: normal;
}
.navbar-brand .author:hover {
text-decoration: underline;
}
.navbar-links {
display: flex;
gap: 20px;
}
.navbar-links a {
color: #ecf0f1;
text-decoration: none;
font-size: 14px;
padding: 8px 12px;
border-radius: 4px;
transition: background-color 0.2s;
}
.navbar-links a:hover {
background: rgba(255,255,255,0.1);
}
.navbar-links a.active {
background: rgba(255,255,255,0.15);
}
/* Main content */
.main-content {
padding: 20px;
}
.container {
width: 90%;
max-width: 1600px;
margin: 0 auto;
background: white;
padding: 30px;
border-radius: 8px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
}
h1 {
margin: 0 0 10px 0;
color: #2c3e50;
font-family: monospace;
font-size: 28px;
}
.subtitle {
color: #7f8c8d;
margin-bottom: 30px;
}
.info-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
gap: 20px;
margin-bottom: 30px;
}
.info-card {
background: #f8f9fa;
padding: 20px;
border-radius: 6px;
border-left: 4px solid #3498db;
}
.info-label {
font-size: 14px;
color: #7f8c8d;
margin-bottom: 5px;
}
.info-value {
font-size: 24px;
font-weight: bold;
color: #2c3e50;
}
.routes-section {
margin-top: 30px;
}
.routes-header {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: 20px;
}
.routes-header h2 {
margin: 0;
color: #2c3e50;
}
.route-count {
background: #e74c3c;
color: white;
padding: 5px 12px;
border-radius: 20px;
font-size: 14px;
font-weight: bold;
}
.route-table {
width: 100%;
border-collapse: collapse;
background: white;
border: 1px solid #e0e0e0;
border-radius: 6px;
overflow: hidden;
}
.route-table th {
background: #34495e;
color: white;
padding: 12px;
text-align: left;
font-weight: 600;
}
.route-table td {
padding: 12px;
border-bottom: 1px solid #e0e0e0;
white-space: nowrap;
}
.route-table tr:hover {
background: #f8f9fa;
}
.route-table tr:last-child td {
border-bottom: none;
}
.as-link {
color: #3498db;
text-decoration: none;
}
.as-link:hover {
text-decoration: underline;
}
.peer-ip {
font-family: monospace;
font-size: 14px;
color: #555;
}
.as-path {
font-family: monospace;
font-size: 13px;
color: #666;
max-width: 600px;
word-wrap: break-word;
white-space: normal !important;
line-height: 1.5;
}
.as-path .as-link {
font-weight: 600;
}
.age {
color: #7f8c8d;
font-size: 14px;
}
.nav-link {
display: inline-block;
margin-bottom: 20px;
color: #3498db;
text-decoration: none;
}
.nav-link:hover {
text-decoration: underline;
}
.origins-section {
margin-top: 30px;
background: #f8f9fa;
padding: 20px;
border-radius: 6px;
}
.origins-section h3 {
margin-top: 0;
color: #2c3e50;
}
.origin-list {
display: flex;
flex-wrap: wrap;
gap: 10px;
}
.origin-item {
background: white;
padding: 10px 15px;
border-radius: 4px;
border: 1px solid #e0e0e0;
}
.empty-state {
text-align: center;
padding: 40px;
color: #7f8c8d;
}
@media (max-width: 768px) {
.container {
padding: 20px;
}
.info-grid {
grid-template-columns: 1fr;
}
.route-table {
font-size: 14px;
}
.as-path {
max-width: 100%;
}
}
</style>
</head>
<body>
<nav class="navbar">
<div class="navbar-brand">
<a href="/">routewatch</a>
<span class="by">by</span>
<a href="https://sneak.berlin" class="author">@sneak</a>
</div>
<div class="navbar-links">
<a href="/">Home</a>
<a href="/status">Status</a>
</div>
</nav>
<main class="main-content">
<div class="container">
<h1>{{.Prefix}}</h1>
<p class="subtitle">IPv{{.IPVersion}} Prefix{{if .MaskLength}} • /{{.MaskLength}}{{end}}</p>
<div class="info-grid">
<div class="info-card">
<div class="info-label">Seen from Peers</div>
<div class="info-value">{{.PeerCount}}</div>
</div>
<div class="info-card">
<div class="info-label">Origin ASNs</div>
<div class="info-value">{{.OriginCount}}</div>
</div>
<div class="info-card">
<div class="info-label">IP Version</div>
<div class="info-value">IPv{{.IPVersion}}</div>
</div>
</div>
{{if .Origins}}
<div class="origins-section">
<h3>Origin ASNs</h3>
<div class="origin-list">
{{range .Origins}}
<div class="origin-item">
<a href="/as/{{.ASN}}" class="as-link">AS{{.ASN}}</a>
{{if .Handle}} ({{.Handle}}){{end}}
{{if .Description}} {{.Description}}{{end}}
<span style="color: #7f8c8d; margin-left: 10px;">{{.PeerCount}} peer{{if ne .PeerCount 1}}s{{end}}</span>
</div>
{{end}}
</div>
</div>
{{end}}
{{if .Routes}}
<div class="routes-section">
<div class="routes-header">
<h2>Route Details</h2>
<span class="route-count">{{.PeerCount}} route{{if ne .PeerCount 1}}s{{end}}</span>
</div>
<table class="route-table">
<thead>
<tr>
<th>Origin AS</th>
<th>Peer IP</th>
<th>AS Path</th>
<th>Next Hop</th>
<th>Last Updated</th>
<th>Age</th>
</tr>
</thead>
<tbody>
{{range .Routes}}
<tr>
<td>
<a href="/as/{{.OriginASN}}" class="as-link">AS{{.OriginASN}}</a>
</td>
<td class="peer-ip">{{.PeerIP}}</td>
<td class="as-path">{{range $i, $as := .ASPathWithHandle}}{{if $i}} → {{end}}<a href="/as/{{$as.ASN}}" class="as-link">{{if $as.Handle}}{{$as.Handle}}{{else}}AS{{$as.ASN}}{{end}}</a>{{end}}</td>
<td class="peer-ip">{{.NextHop}}</td>
<td>{{.LastUpdated.Format "2006-01-02 15:04:05"}}</td>
<td class="age">{{.LastUpdated | timeSince}}</td>
</tr>
{{end}}
</tbody>
</table>
</div>
{{else}}
<div class="empty-state">
<p>No routes found for this prefix</p>
</div>
{{end}}
</div>
</main>
</body>
</html>
internal/templates/prefix_length.html
+185
View File
@@ -0,0 +1,185 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Prefixes with /{{ .MaskLength }} - RouteWatch</title>
<style>
* {
box-sizing: border-box;
}
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
margin: 0;
padding: 0;
background: #f5f5f5;
}
/* Navbar styles */
.navbar {
background: #2c3e50;
padding: 0 20px;
display: flex;
align-items: center;
justify-content: space-between;
height: 56px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
}
.navbar-brand {
display: flex;
align-items: center;
gap: 8px;
font-size: 18px;
font-weight: 600;
}
.navbar-brand a {
color: white;
text-decoration: none;
}
.navbar-brand a:hover {
color: #ecf0f1;
}
.navbar-brand .by {
font-weight: normal;
color: #95a5a6;
font-size: 14px;
}
.navbar-brand .author {
color: #3498db;
font-weight: normal;
}
.navbar-brand .author:hover {
text-decoration: underline;
}
.navbar-links {
display: flex;
gap: 20px;
}
.navbar-links a {
color: #ecf0f1;
text-decoration: none;
font-size: 14px;
padding: 8px 12px;
border-radius: 4px;
transition: background-color 0.2s;
}
.navbar-links a:hover {
background: rgba(255,255,255,0.1);
}
.navbar-links a.active {
background: rgba(255,255,255,0.15);
}
/* Main content */
.main-content {
max-width: 1200px;
margin: 0 auto;
padding: 20px;
}
h1 {
color: #333;
margin-bottom: 10px;
}
.subtitle {
color: #666;
margin-bottom: 30px;
}
.info-card {
background: white;
padding: 20px;
border-radius: 8px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
margin-bottom: 20px;
}
table {
width: 100%;
border-collapse: collapse;
background: white;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
border-radius: 8px;
overflow: hidden;
}
th {
background: #f8f9fa;
padding: 12px;
text-align: left;
font-weight: 600;
color: #333;
border-bottom: 2px solid #dee2e6;
}
td {
padding: 12px;
border-bottom: 1px solid #eee;
}
tr:last-child td {
border-bottom: none;
}
tr:hover {
background: #f8f9fa;
}
a {
color: #0066cc;
text-decoration: none;
}
a:hover {
text-decoration: underline;
}
.prefix-link {
font-family: 'SF Mono', Monaco, 'Cascadia Mono', 'Roboto Mono', Consolas, 'Courier New', monospace;
}
.as-link {
white-space: nowrap;
}
.age {
color: #666;
font-size: 0.9em;
}
.back-link {
display: inline-block;
margin-bottom: 20px;
color: #0066cc;
}
</style>
</head>
<body>
<nav class="navbar">
<div class="navbar-brand">
<a href="/">routewatch</a>
<span class="by">by</span>
<a href="https://sneak.berlin" class="author">@sneak</a>
</div>
<div class="navbar-links">
<a href="/">Home</a>
<a href="/status">Status</a>
</div>
</nav>
<main class="main-content">
<h1>IPv{{ .IPVersion }} Prefixes with /{{ .MaskLength }}</h1>
<p class="subtitle">Showing {{ .Count }} randomly selected prefixes</p>
<table>
<thead>
<tr>
<th>Prefix</th>
<th>Age</th>
<th>Origin AS</th>
</tr>
</thead>
<tbody>
{{ range .Prefixes }}
<tr>
<td><a href="{{ .Prefix | prefixURL }}" class="prefix-link">{{ .Prefix }}</a></td>
<td class="age">{{ .Age }}</td>
<td>
<a href="/as/{{ .OriginASN }}" class="as-link">
AS{{ .OriginASN }}{{ if .OriginASDescription }} ({{ .OriginASDescription }}){{ end }}
</a>
</td>
</tr>
{{ end }}
</tbody>
</table>
</main>
</body>
</html>
internal/templates/status.html
+412 -19
View File
@@ -5,12 +5,76 @@
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>RouteWatch Status</title>
<style>
* {
box-sizing: border-box;
}
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
margin: 0;
padding: 0;
background: #f5f5f5;
}
/* Navbar styles */
.navbar {
background: #2c3e50;
padding: 0 20px;
display: flex;
align-items: center;
justify-content: space-between;
height: 56px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
}
.navbar-brand {
display: flex;
align-items: center;
gap: 8px;
font-size: 18px;
font-weight: 600;
}
.navbar-brand a {
color: white;
text-decoration: none;
}
.navbar-brand a:hover {
color: #ecf0f1;
}
.navbar-brand .by {
font-weight: normal;
color: #95a5a6;
font-size: 14px;
}
.navbar-brand .author {
color: #3498db;
font-weight: normal;
}
.navbar-brand .author:hover {
text-decoration: underline;
}
.navbar-links {
display: flex;
gap: 20px;
}
.navbar-links a {
color: #ecf0f1;
text-decoration: none;
font-size: 14px;
padding: 8px 12px;
border-radius: 4px;
transition: background-color 0.2s;
}
.navbar-links a:hover {
background: rgba(255,255,255,0.1);
}
.navbar-links a.active {
background: rgba(255,255,255,0.15);
}
/* Main content */
.main-content {
max-width: 1200px;
margin: 0 auto;
padding: 20px;
background: #f5f5f5;
}
h1 {
color: #333;
@@ -49,6 +113,16 @@
font-family: 'SF Mono', Monaco, 'Cascadia Mono', 'Roboto Mono', Consolas, 'Courier New', monospace;
color: #333;
}
.metric-value.metric-link {
text-decoration: underline;
text-decoration-style: dashed;
text-underline-offset: 2px;
cursor: pointer;
}
.metric-value.metric-link:hover {
color: #0066cc;
text-decoration-style: solid;
}
.connected {
color: #22c55e;
}
@@ -62,14 +136,47 @@
border-radius: 4px;
margin-top: 20px;
}
.footer {
margin-top: 40px;
padding: 20px;
background: white;
border-radius: 8px;
box-shadow: 0 -2px 10px rgba(0,0,0,0.1);
text-align: center;
color: #666;
font-size: 14px;
}
.footer a {
color: #0066cc;
text-decoration: none;
}
.footer a:hover {
text-decoration: underline;
}
.footer .separator {
margin: 0 10px;
color: #ccc;
}
</style>
</head>
<body>
<h1>RouteWatch Status</h1>
<nav class="navbar">
<div class="navbar-brand">
<a href="/">routewatch</a>
<span class="by">by</span>
<a href="https://sneak.berlin" class="author">@sneak</a>
</div>
<div class="navbar-links">
<a href="/">Home</a>
<a href="/status" class="active">Status</a>
</div>
</nav>
<main class="main-content">
<div id="error" class="error" style="display: none;"></div>
<div class="status-grid">
<div class="status-card">
<h2>Connection Status</h2>
<h2>RouteWatch</h2>
<div class="metric">
<span class="metric-label">Status</span>
<span class="metric-value" id="connected">-</span>
@@ -78,10 +185,34 @@
<span class="metric-label">Uptime</span>
<span class="metric-value" id="uptime">-</span>
</div>
<div class="metric">
<span class="metric-label">Go Version</span>
<span class="metric-value" id="go_version">-</span>
</div>
<div class="metric">
<span class="metric-label">Goroutines</span>
<span class="metric-value" id="goroutines">-</span>
</div>
<div class="metric">
<span class="metric-label">Memory Usage</span>
<span class="metric-value" id="memory_usage">-</span>
</div>
</div>
<div class="status-card">
<h2>Stream Statistics</h2>
<div class="metric">
<span class="metric-label">Connection Duration</span>
<span class="metric-value" id="connection_duration">-</span>
</div>
<div class="metric">
<span class="metric-label">Reconnections</span>
<span class="metric-value" id="reconnect_count">-</span>
</div>
<div class="metric">
<span class="metric-label">BGP Peers</span>
<span class="metric-value" id="bgp_peer_count">-</span>
</div>
<div class="metric">
<span class="metric-label">Total Messages</span>
<span class="metric-value" id="total_messages">-</span>
@@ -90,13 +221,49 @@
<span class="metric-label">Messages/sec</span>
<span class="metric-value" id="messages_per_sec">-</span>
</div>
<div class="metric">
<span class="metric-label">Announcements</span>
<span class="metric-value" id="announcements">-</span>
</div>
<div class="metric">
<span class="metric-label">Withdrawals</span>
<span class="metric-value" id="withdrawals">-</span>
</div>
<div class="metric">
<span class="metric-label">Route Churn/sec</span>
<span class="metric-value" id="route_churn_per_sec">-</span>
</div>
<div class="metric">
<span class="metric-label">Total Data</span>
<span class="metric-value" id="total_bytes">-</span>
<span class="metric-value" id="total_wire_bytes">-</span>
</div>
<div class="metric">
<span class="metric-label">Throughput</span>
<span class="metric-value" id="mbits_per_sec">-</span>
<span class="metric-value" id="wire_mbits_per_sec">-</span>
</div>
</div>
<div class="status-card">
<h2>GC Statistics</h2>
<div class="metric">
<span class="metric-label">GC Runs</span>
<span class="metric-value" id="gc_num">-</span>
</div>
<div class="metric">
<span class="metric-label">Total Pause</span>
<span class="metric-value" id="gc_total_pause">-</span>
</div>
<div class="metric">
<span class="metric-label">Last Pause</span>
<span class="metric-value" id="gc_last_pause">-</span>
</div>
<div class="metric">
<span class="metric-label">Heap Alloc</span>
<span class="metric-value" id="gc_heap_alloc">-</span>
</div>
<div class="metric">
<span class="metric-label">Heap Sys</span>
<span class="metric-value" id="gc_heap_sys">-</span>
</div>
</div>
@@ -110,18 +277,14 @@
<span class="metric-label">Total Prefixes</span>
<span class="metric-value" id="prefixes">-</span>
</div>
<div class="metric">
<span class="metric-label">IPv4 Prefixes</span>
<span class="metric-value" id="ipv4_prefixes">-</span>
</div>
<div class="metric">
<span class="metric-label">IPv6 Prefixes</span>
<span class="metric-value" id="ipv6_prefixes">-</span>
</div>
<div class="metric">
<span class="metric-label">Peerings</span>
<span class="metric-value" id="peerings">-</span>
</div>
<div class="metric">
<span class="metric-label">Peers</span>
<span class="metric-value" id="peers">-</span>
</div>
<div class="metric">
<span class="metric-label">Database Size</span>
<span class="metric-value" id="database_size">-</span>
@@ -134,6 +297,14 @@
<span class="metric-label">Live Routes</span>
<span class="metric-value" id="live_routes">-</span>
</div>
<div class="metric">
<span class="metric-label">IPv4 Prefixes</span>
<span class="metric-value" id="ipv4_prefixes">-</span>
</div>
<div class="metric">
<span class="metric-label">IPv6 Prefixes</span>
<span class="metric-value" id="ipv6_prefixes">-</span>
</div>
<div class="metric">
<span class="metric-label">IPv4 Routes</span>
<span class="metric-value" id="ipv4_routes">-</span>
@@ -150,6 +321,62 @@
<span class="metric-label">IPv6 Updates/sec</span>
<span class="metric-value" id="ipv6_updates_per_sec">-</span>
</div>
<div class="metric">
<span class="metric-label">Oldest Route</span>
<span class="metric-value" id="oldest_route">-</span>
</div>
<div class="metric">
<span class="metric-label">Newest Route</span>
<span class="metric-value" id="newest_route">-</span>
</div>
</div>
<div class="status-card">
<h2>WHOIS Fetcher</h2>
<div class="metric">
<span class="metric-label">Fresh ASNs</span>
<span class="metric-value" id="whois_fresh">-</span>
</div>
<div class="metric">
<span class="metric-label">Stale ASNs</span>
<span class="metric-value" id="whois_stale">-</span>
</div>
<div class="metric">
<span class="metric-label">Never Fetched</span>
<span class="metric-value" id="whois_never">-</span>
</div>
<div class="metric">
<span class="metric-label">Fresh %</span>
<span class="metric-value" id="whois_percent">-</span>
</div>
<div class="metric">
<span class="metric-label">Successes (1h)</span>
<span class="metric-value" id="whois_successes">-</span>
</div>
<div class="metric">
<span class="metric-label">Errors (1h)</span>
<span class="metric-value" id="whois_errors">-</span>
</div>
<div class="metric">
<span class="metric-label">Current Interval</span>
<span class="metric-value" id="whois_interval">-</span>
</div>
</div>
</div>
<div class="status-grid">
<div class="status-card">
<h2>IPv4 Prefix Distribution</h2>
<div id="ipv4-prefix-distribution">
<!-- Will be populated dynamically -->
</div>
</div>
<div class="status-card">
<h2>IPv6 Prefix Distribution</h2>
<div id="ipv6-prefix-distribution">
<!-- Will be populated dynamically -->
</div>
</div>
</div>
@@ -170,6 +397,61 @@
return num.toLocaleString();
}
function formatProcessingTime(ms) {
if (ms < 0.001) {
return (ms * 1000).toFixed(0) + ' µs';
} else if (ms < 0.01) {
return (ms * 1000).toFixed(1) + ' µs';
} else if (ms < 1) {
return ms.toFixed(3) + ' ms';
} else {
return ms.toFixed(2) + ' ms';
}
}
function formatRelativeTime(isoString) {
if (!isoString) return '-';
const date = new Date(isoString);
const now = new Date();
const diffMs = now - date;
const diffSec = Math.floor(diffMs / 1000);
const diffMin = Math.floor(diffSec / 60);
const diffHour = Math.floor(diffMin / 60);
const diffDay = Math.floor(diffHour / 24);
if (diffSec < 60) return diffSec + 's ago';
if (diffMin < 60) return diffMin + 'm ago';
if (diffHour < 24) return diffHour + 'h ' + (diffMin % 60) + 'm ago';
return diffDay + 'd ' + (diffHour % 24) + 'h ago';
}
function updatePrefixDistribution(elementId, distribution) {
const container = document.getElementById(elementId);
container.innerHTML = '';
if (!distribution || distribution.length === 0) {
container.innerHTML = '<div class="metric"><span class="metric-label">No data</span></div>';
return;
}
// Sort by mask length
distribution.sort((a, b) => a.mask_length - b.mask_length);
// Determine the URL path based on whether this is IPv4 or IPv6
const isIPv6 = elementId.includes('ipv6');
const urlPath = isIPv6 ? '/prefixlength6/' : '/prefixlength/';
distribution.forEach(item => {
const metric = document.createElement('div');
metric.className = 'metric';
metric.innerHTML = `
<span class="metric-label">/${item.mask_length}</span>
<a href="${urlPath}${item.mask_length}" class="metric-value metric-link">${formatNumber(item.count)}</a>
`;
container.appendChild(metric);
});
}
function updateHandlerStats(handlerStats) {
const container = document.getElementById('handler-stats-container');
container.innerHTML = '';
@@ -187,6 +469,10 @@
<span class="metric-label">Queue</span>
<span class="metric-value">${handler.queue_length}/${handler.queue_capacity}</span>
</div>
<div class="metric">
<span class="metric-label">High Water Mark</span>
<span class="metric-value">${handler.queue_high_water_mark}/${handler.queue_capacity} (${Math.round(handler.queue_high_water_mark * 100 / handler.queue_capacity)}%)</span>
</div>
<div class="metric">
<span class="metric-label">Processed</span>
<span class="metric-value">${formatNumber(handler.processed_count)}</span>
@@ -197,11 +483,11 @@
</div>
<div class="metric">
<span class="metric-label">Avg Time</span>
<span class="metric-value">${handler.avg_process_time_ms.toFixed(2)} ms</span>
<span class="metric-value">${formatProcessingTime(handler.avg_process_time_ms)}</span>
</div>
<div class="metric">
<span class="metric-label">Min/Max Time</span>
<span class="metric-value">${handler.min_process_time_ms.toFixed(2)} / ${handler.max_process_time_ms.toFixed(2)} ms</span>
<span class="metric-value">${formatProcessingTime(handler.min_process_time_ms)} / ${formatProcessingTime(handler.max_process_time_ms)}</span>
</div>
`;
@@ -209,6 +495,60 @@
});
}
function resetAllFields() {
// Reset all metric fields to '-'
document.getElementById('connected').textContent = '-';
document.getElementById('connected').className = 'metric-value';
document.getElementById('uptime').textContent = '-';
document.getElementById('go_version').textContent = '-';
document.getElementById('goroutines').textContent = '-';
document.getElementById('memory_usage').textContent = '-';
document.getElementById('connection_duration').textContent = '-';
document.getElementById('reconnect_count').textContent = '-';
document.getElementById('bgp_peer_count').textContent = '-';
document.getElementById('total_messages').textContent = '-';
document.getElementById('messages_per_sec').textContent = '-';
document.getElementById('announcements').textContent = '-';
document.getElementById('withdrawals').textContent = '-';
document.getElementById('route_churn_per_sec').textContent = '-';
document.getElementById('total_wire_bytes').textContent = '-';
document.getElementById('wire_mbits_per_sec').textContent = '-';
document.getElementById('gc_num').textContent = '-';
document.getElementById('gc_total_pause').textContent = '-';
document.getElementById('gc_last_pause').textContent = '-';
document.getElementById('gc_heap_alloc').textContent = '-';
document.getElementById('gc_heap_sys').textContent = '-';
document.getElementById('asns').textContent = '-';
document.getElementById('prefixes').textContent = '-';
document.getElementById('ipv4_prefixes').textContent = '-';
document.getElementById('ipv6_prefixes').textContent = '-';
document.getElementById('peerings').textContent = '-';
document.getElementById('peers').textContent = '-';
document.getElementById('database_size').textContent = '-';
document.getElementById('live_routes').textContent = '-';
document.getElementById('ipv4_routes').textContent = '-';
document.getElementById('ipv6_routes').textContent = '-';
document.getElementById('ipv4_updates_per_sec').textContent = '-';
document.getElementById('ipv6_updates_per_sec').textContent = '-';
document.getElementById('oldest_route').textContent = '-';
document.getElementById('newest_route').textContent = '-';
document.getElementById('whois_fresh').textContent = '-';
document.getElementById('whois_stale').textContent = '-';
document.getElementById('whois_never').textContent = '-';
document.getElementById('whois_percent').textContent = '-';
document.getElementById('whois_successes').textContent = '-';
document.getElementById('whois_errors').textContent = '-';
document.getElementById('whois_errors').className = 'metric-value';
document.getElementById('whois_interval').textContent = '-';
// Clear handler stats
document.getElementById('handler-stats-container').innerHTML = '';
// Clear prefix distributions
document.getElementById('ipv4-prefix-distribution').innerHTML = '<div class="metric"><span class="metric-label">No data</span></div>';
document.getElementById('ipv6-prefix-distribution').innerHTML = '<div class="metric"><span class="metric-label">No data</span></div>';
}
function updateStatus() {
fetch('/api/v1/stats')
.then(response => response.json())
@@ -217,6 +557,7 @@
if (response.status === 'error') {
document.getElementById('error').textContent = 'Error: ' + response.error.msg;
document.getElementById('error').style.display = 'block';
resetAllFields();
return;
}
@@ -230,37 +571,89 @@
// Update all metrics
document.getElementById('uptime').textContent = data.uptime;
document.getElementById('go_version').textContent = data.go_version;
document.getElementById('goroutines').textContent = formatNumber(data.goroutines);
document.getElementById('memory_usage').textContent = data.memory_usage;
document.getElementById('connection_duration').textContent = data.connection_duration;
document.getElementById('reconnect_count').textContent = formatNumber(data.reconnect_count);
document.getElementById('total_messages').textContent = formatNumber(data.total_messages);
document.getElementById('messages_per_sec').textContent = data.messages_per_sec.toFixed(1);
document.getElementById('total_bytes').textContent = formatBytes(data.total_bytes);
document.getElementById('mbits_per_sec').textContent = data.mbits_per_sec.toFixed(2) + ' Mbps';
document.getElementById('total_wire_bytes').textContent = formatBytes(data.total_wire_bytes);
document.getElementById('wire_mbits_per_sec').textContent = data.wire_mbits_per_sec.toFixed(2) + ' Mbps';
document.getElementById('asns').textContent = formatNumber(data.asns);
document.getElementById('prefixes').textContent = formatNumber(data.prefixes);
document.getElementById('ipv4_prefixes').textContent = formatNumber(data.ipv4_prefixes);
document.getElementById('ipv6_prefixes').textContent = formatNumber(data.ipv6_prefixes);
document.getElementById('peerings').textContent = formatNumber(data.peerings);
document.getElementById('peers').textContent = formatNumber(data.peers);
document.getElementById('database_size').textContent = formatBytes(data.database_size_bytes);
document.getElementById('live_routes').textContent = formatNumber(data.live_routes);
document.getElementById('ipv4_routes').textContent = formatNumber(data.ipv4_routes);
document.getElementById('ipv6_routes').textContent = formatNumber(data.ipv6_routes);
document.getElementById('ipv4_updates_per_sec').textContent = data.ipv4_updates_per_sec.toFixed(1);
document.getElementById('ipv6_updates_per_sec').textContent = data.ipv6_updates_per_sec.toFixed(1);
document.getElementById('oldest_route').textContent = formatRelativeTime(data.oldest_route);
document.getElementById('newest_route').textContent = formatRelativeTime(data.newest_route);
// Update stream stats
if (data.stream) {
document.getElementById('bgp_peer_count').textContent = formatNumber(data.stream.bgp_peer_count);
document.getElementById('announcements').textContent = formatNumber(data.stream.announcements);
document.getElementById('withdrawals').textContent = formatNumber(data.stream.withdrawals);
document.getElementById('route_churn_per_sec').textContent = data.stream.route_churn_per_sec.toFixed(1);
}
// Update GC stats
if (data.gc) {
document.getElementById('gc_num').textContent = formatNumber(data.gc.num_gc);
document.getElementById('gc_total_pause').textContent = data.gc.total_pause_ms + ' ms';
document.getElementById('gc_last_pause').textContent = data.gc.last_pause_ms.toFixed(3) + ' ms';
document.getElementById('gc_heap_alloc').textContent = formatBytes(data.gc.heap_alloc_bytes);
document.getElementById('gc_heap_sys').textContent = formatBytes(data.gc.heap_sys_bytes);
}
// Update WHOIS stats
if (data.whois_stats) {
document.getElementById('whois_fresh').textContent = formatNumber(data.whois_stats.fresh_asns);
document.getElementById('whois_stale').textContent = formatNumber(data.whois_stats.stale_asns);
document.getElementById('whois_never').textContent = formatNumber(data.whois_stats.never_fetched);
document.getElementById('whois_percent').textContent = data.whois_stats.fresh_percent.toFixed(1) + '%';
document.getElementById('whois_successes').textContent = formatNumber(data.whois_stats.successes_last_hour);
const errorsEl = document.getElementById('whois_errors');
errorsEl.textContent = formatNumber(data.whois_stats.errors_last_hour);
errorsEl.className = 'metric-value' + (data.whois_stats.errors_last_hour > 0 ? ' disconnected' : '');
document.getElementById('whois_interval').textContent = data.whois_stats.current_interval;
}
// Update handler stats
updateHandlerStats(data.handler_stats || []);
// Update prefix distribution
updatePrefixDistribution('ipv4-prefix-distribution', data.ipv4_prefix_distribution);
updatePrefixDistribution('ipv6-prefix-distribution', data.ipv6_prefix_distribution);
// Clear any errors
document.getElementById('error').style.display = 'none';
})
.catch(error => {
document.getElementById('error').textContent = 'Error fetching status: ' + error;
document.getElementById('error').style.display = 'block';
resetAllFields();
});
}
// Update immediately and then every 500ms
// Update immediately and then every 2 seconds
updateStatus();
setInterval(updateStatus, 500);
setInterval(updateStatus, 2000);
</script>
</main>
<footer class="footer">
<span><a href="{{appRepoURL}}">{{appName}}</a> by <a href="{{appAuthorURL}}">{{appAuthor}}</a></span>
<span class="separator">|</span>
<span>{{appLicense}}</span>
<span class="separator">|</span>
<span><a href="{{appGitCommitURL}}">{{appGitRevision}}</a></span>
</footer>
</body>
</html>
internal/templates/templates.go
+145 -1
View File
@@ -4,15 +4,41 @@ package templates
import (
_ "embed"
"html/template"
"net/url"
"strings"
"sync"
"time"
"git.eeqj.de/sneak/routewatch/internal/version"
)
//go:embed index.html
var indexHTML string
//go:embed status.html
var statusHTML string
//go:embed as_detail.html
var asDetailHTML string
//go:embed prefix_detail.html
var prefixDetailHTML string
//go:embed prefix_length.html
var prefixLengthHTML string
// Templates contains all parsed templates
type Templates struct {
// Index is the template for the home page
Index *template.Template
// Status is the template for the main status page
Status *template.Template
// ASDetail is the template for displaying AS (Autonomous System) details
ASDetail *template.Template
// PrefixDetail is the template for displaying prefix details
PrefixDetail *template.Template
// PrefixLength is the template for displaying prefixes by length
PrefixLength *template.Template
}
var (
@@ -22,17 +48,115 @@ var (
once sync.Once
)
const (
hoursPerDay = 24
daysPerMonth = 30
cidrPartCount = 2 // A CIDR has two parts: prefix and length
)
// timeSince returns a human-readable duration since the given time
func timeSince(t time.Time) string {
duration := time.Since(t)
if duration < time.Minute {
return "just now"
}
if duration < time.Hour {
minutes := int(duration.Minutes())
if minutes == 1 {
return "1 minute ago"
}
return duration.Truncate(time.Minute).String() + " ago"
}
if duration < hoursPerDay*time.Hour {
hours := int(duration.Hours())
if hours == 1 {
return "1 hour ago"
}
return duration.Truncate(time.Hour).String() + " ago"
}
days := int(duration.Hours() / hoursPerDay)
if days == 1 {
return "1 day ago"
}
if days < daysPerMonth {
return duration.Truncate(hoursPerDay*time.Hour).String() + " ago"
}
return t.Format("2006-01-02")
}
// prefixURL generates a URL path for a prefix in CIDR notation.
// Takes a prefix like "192.168.1.0/24" and returns "/prefix/192.168.1.0/24"
// with the prefix part URL-encoded to handle IPv6 colons.
func prefixURL(cidr string) string {
// Split CIDR into prefix and length
parts := strings.SplitN(cidr, "/", cidrPartCount)
if len(parts) != cidrPartCount {
// Fallback if no slash found
return "/prefix/" + url.PathEscape(cidr) + "/0"
}
return "/prefix/" + url.PathEscape(parts[0]) + "/" + parts[1]
}
// initTemplates parses all embedded templates
func initTemplates() {
var err error
defaultTemplates = &Templates{}
// Create common template functions
funcs := template.FuncMap{
"timeSince": timeSince,
"urlEncode": url.QueryEscape,
"prefixURL": prefixURL,
"appName": func() string { return version.Name },
"appAuthor": func() string { return version.Author },
"appAuthorURL": func() string { return version.AuthorURL },
"appLicense": func() string { return version.License },
"appRepoURL": func() string { return version.RepoURL },
"appGitRevision": func() string { return version.GitRevisionShort },
"appGitCommitURL": version.CommitURL,
}
// Parse index template
defaultTemplates.Index, err = template.New("index").Funcs(funcs).Parse(indexHTML)
if err != nil {
panic("failed to parse index template: " + err.Error())
}
// Parse status template
defaultTemplates.Status, err = template.New("status").Parse(statusHTML)
defaultTemplates.Status, err = template.New("status").Funcs(funcs).Parse(statusHTML)
if err != nil {
panic("failed to parse status template: " + err.Error())
}
// Parse AS detail template
defaultTemplates.ASDetail, err = template.New("asDetail").
Funcs(funcs).Parse(asDetailHTML)
if err != nil {
panic("failed to parse AS detail template: " + err.Error())
}
// Parse prefix detail template
defaultTemplates.PrefixDetail, err = template.New("prefixDetail").
Funcs(funcs).Parse(prefixDetailHTML)
if err != nil {
panic("failed to parse prefix detail template: " + err.Error())
}
// Parse prefix length template
defaultTemplates.PrefixLength, err = template.New("prefixLength").
Funcs(funcs).Parse(prefixLengthHTML)
if err != nil {
panic("failed to parse prefix length template: " + err.Error())
}
}
// Get returns the singleton Templates instance
@@ -42,7 +166,27 @@ func Get() *Templates {
return defaultTemplates
}
// IndexTemplate returns the parsed index template
func IndexTemplate() *template.Template {
return Get().Index
}
// StatusTemplate returns the parsed status template
func StatusTemplate() *template.Template {
return Get().Status
}
// ASDetailTemplate returns the parsed AS detail template
func ASDetailTemplate() *template.Template {
return Get().ASDetail
}
// PrefixDetailTemplate returns the parsed prefix detail template
func PrefixDetailTemplate() *template.Template {
return Get().PrefixDetail
}
// PrefixLengthTemplate returns the parsed prefix length template
func PrefixLengthTemplate() *template.Template {
return Get().PrefixLength
}
internal/version/version.go
+35
View File
@@ -0,0 +1,35 @@
// Package version provides build version information
package version
// Build-time variables set via ldflags
//
//nolint:gochecknoglobals // variables so that ldflags can set them at build time
var (
// GitRevision is the git commit hash
GitRevision = "unknown"
// GitRevisionShort is the version the page footer shows: the tag or
// short commit hash from `git describe --tags --always`
GitRevisionShort = "unknown"
)
const (
// Name is the program name
Name = "routewatch"
// Author is the program author
Author = "@sneak"
// AuthorURL is the author's website
AuthorURL = "https://sneak.berlin"
// License is the program license
License = "WTFPL"
// RepoURL is the git repository URL
RepoURL = "https://git.eeqj.de/sneak/routewatch"
)
// CommitURL returns the URL to view the current commit
func CommitURL() string {
if GitRevision == "unknown" {
return RepoURL
}
return RepoURL + "/commit/" + GitRevision
}
internal/whois/whois.go
+375
View File
@@ -0,0 +1,375 @@
// Package whois provides WHOIS lookup functionality for ASN information.
package whois
import (
"bufio"
"context"
"fmt"
"net"
"regexp"
"strings"
"time"
)
// Timeout constants for WHOIS queries.
const (
dialTimeout = 10 * time.Second
readTimeout = 30 * time.Second
writeTimeout = 5 * time.Second
)
// Parsing constants.
const (
keyValueParts = 2 // Expected parts when splitting "key: value"
lacnicDateFormatLen = 8 // Length of YYYYMMDD date format
)
// WHOIS server addresses.
const (
whoisServerIANA = "whois.iana.org:43"
whoisServerARIN = "whois.arin.net:43"
whoisServerRIPE = "whois.ripe.net:43"
whoisServerAPNIC = "whois.apnic.net:43"
whoisServerLACNIC = "whois.lacnic.net:43"
whoisServerAFRINIC = "whois.afrinic.net:43"
)
// RIR identifiers.
const (
RIRARIN = "ARIN"
RIRRIPE = "RIPE"
RIRAPNIC = "APNIC"
RIRLACNIC = "LACNIC"
RIRAFRNIC = "AFRINIC"
)
// ASNInfo contains parsed WHOIS information for an ASN.
type ASNInfo struct {
ASN int
ASName string
OrgName string
OrgID string
Address string
CountryCode string
AbuseEmail string
AbusePhone string
TechEmail string
TechPhone string
RIR string
RegDate *time.Time
LastMod *time.Time
RawResponse string
}
// Client performs WHOIS lookups for ASNs.
type Client struct {
// Dialer for creating connections (can be overridden for testing)
dialer *net.Dialer
}
// NewClient creates a new WHOIS client.
func NewClient() *Client {
return &Client{
dialer: &net.Dialer{
Timeout: dialTimeout,
},
}
}
// LookupASN queries WHOIS for the given ASN and returns parsed information.
func (c *Client) LookupASN(ctx context.Context, asn int) (*ASNInfo, error) {
// Query IANA first to find the authoritative RIR
query := fmt.Sprintf("AS%d", asn)
ianaResp, err := c.query(ctx, whoisServerIANA, query)
if err != nil {
return nil, fmt.Errorf("IANA query failed: %w", err)
}
// Determine RIR from IANA response
rir, whoisServer := c.parseIANAReferral(ianaResp)
if whoisServer == "" {
// No referral, try to parse what we have
return c.parseResponse(asn, rir, ianaResp), nil
}
// Query the authoritative RIR
rirResp, err := c.query(ctx, whoisServer, query)
if err != nil {
// Return partial data from IANA if RIR query fails
info := c.parseResponse(asn, rir, ianaResp)
info.RawResponse = fmt.Sprintf("%s\n--- RIR query failed: %v ---\n",
ianaResp, err)
return info, nil
}
// Combine responses and parse
fullResponse := ianaResp + "\n" + rirResp
info := c.parseResponse(asn, rir, fullResponse)
info.RawResponse = fullResponse
return info, nil
}
// query performs a raw WHOIS query to the specified server.
func (c *Client) query(ctx context.Context, server, query string) (string, error) {
conn, err := c.dialer.DialContext(ctx, "tcp", server)
if err != nil {
return "", fmt.Errorf("dial %s: %w", server, err)
}
defer func() { _ = conn.Close() }()
// Set deadlines
err = conn.SetWriteDeadline(time.Now().Add(writeTimeout))
if err != nil {
return "", fmt.Errorf("set write deadline: %w", err)
}
// Send query
_, err = fmt.Fprintf(conn, "%s\r\n", query)
if err != nil {
return "", fmt.Errorf("write query: %w", err)
}
// Read response
err = conn.SetReadDeadline(time.Now().Add(readTimeout))
if err != nil {
return "", fmt.Errorf("set read deadline: %w", err)
}
var sb strings.Builder
scanner := bufio.NewScanner(conn)
for scanner.Scan() {
sb.WriteString(scanner.Text())
sb.WriteString("\n")
}
err = scanner.Err()
if err != nil {
return sb.String(), fmt.Errorf("read response: %w", err)
}
return sb.String(), nil
}
// parseIANAReferral extracts the RIR and WHOIS server from an IANA response.
func (c *Client) parseIANAReferral(response string) (string, string) {
var rir string
for line := range strings.SplitSeq(response, "\n") {
line = strings.TrimSpace(line)
// Look for "refer:" line
if strings.HasPrefix(strings.ToLower(line), "refer:") {
server := strings.TrimSpace(strings.TrimPrefix(line, "refer:"))
server = strings.TrimSpace(strings.TrimPrefix(server, "Refer:"))
switch {
case strings.Contains(server, "arin"):
return RIRARIN, whoisServerARIN
case strings.Contains(server, "ripe"):
return RIRRIPE, whoisServerRIPE
case strings.Contains(server, "apnic"):
return RIRAPNIC, whoisServerAPNIC
case strings.Contains(server, "lacnic"):
return RIRLACNIC, whoisServerLACNIC
case strings.Contains(server, "afrinic"):
return RIRAFRNIC, whoisServerAFRINIC
default:
// Unknown server, add port if missing
if !strings.Contains(server, ":") {
server += ":43"
}
return "", server
}
}
// Also check organisation line for RIR hints
if strings.HasPrefix(strings.ToLower(line), "organisation:") {
orgRIR := rirInOrganisationLine(strings.ToLower(line))
if orgRIR != "" {
rir = orgRIR
}
}
}
return rir, ""
}
// rirInOrganisationLine returns the RIR a lowercased IANA "organisation:"
// line names, or "" when it names none.
func rirInOrganisationLine(org string) string {
switch {
case strings.Contains(org, "arin"):
return RIRARIN
case strings.Contains(org, "ripe"):
return RIRRIPE
case strings.Contains(org, "apnic"):
return RIRAPNIC
case strings.Contains(org, "lacnic"):
return RIRLACNIC
case strings.Contains(org, "afrinic"):
return RIRAFRNIC
default:
return ""
}
}
// parseResponse extracts ASN information from a WHOIS response.
func (c *Client) parseResponse(asn int, rir, response string) *ASNInfo {
info := &ASNInfo{
ASN: asn,
RIR: rir,
RawResponse: response,
}
lines := strings.Split(response, "\n")
var addressLines []string
for _, line := range lines {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "%") ||
strings.HasPrefix(line, "#") {
continue
}
// Split on first colon
parts := strings.SplitN(line, ":", keyValueParts)
if len(parts) != keyValueParts {
continue
}
key := strings.TrimSpace(strings.ToLower(parts[0]))
value := strings.TrimSpace(parts[1])
if value == "" {
continue
}
// Address (collect multiple lines)
if key == "address" {
addressLines = append(addressLines, value)
continue
}
c.setResponseField(info, key, value)
}
// Combine address lines
if len(addressLines) > 0 {
info.Address = strings.Join(addressLines, "\n")
}
// Extract abuse email from comment lines (common in ARIN responses)
if info.AbuseEmail == "" {
info.AbuseEmail = c.extractAbuseEmail(response)
}
return info
}
// setResponseField stores the value of one "key: value" line of a WHOIS
// response in info, keeping the first value seen for each field.
func (c *Client) setResponseField(info *ASNInfo, key, value string) {
switch key {
// Country
case "country":
if info.CountryCode == "" && len(value) == 2 {
info.CountryCode = strings.ToUpper(value)
}
// Registration dates
case "regdate", "created":
if info.RegDate == nil {
info.RegDate = c.parseDate(value)
}
case "updated", "last-modified", "changed":
if info.LastMod == nil {
info.LastMod = c.parseDate(value)
}
default:
field := textFieldForKey(info, key)
if field != nil && *field == "" {
*field = value
}
}
}
// textFieldForKey returns the text field of info that a WHOIS response key
// fills, or nil when the key fills none.
func textFieldForKey(info *ASNInfo, key string) *string {
switch key {
// AS Name (varies by RIR)
case "asname", "as-name":
return &info.ASName
// Organization
case "orgname", "org-name", "owner":
return &info.OrgName
case "orgid", "org-id", "org":
return &info.OrgID
// Abuse contact
case "orgabuseemail", "abuse-mailbox":
return &info.AbuseEmail
case "orgabusephone":
return &info.AbusePhone
// Tech contact
case "orgtechemail":
return &info.TechEmail
case "orgtechphone":
return &info.TechPhone
default:
return nil
}
}
// parseDate attempts to parse various date formats used in WHOIS responses.
func (c *Client) parseDate(value string) *time.Time {
// Common formats
formats := []string{
"2006-01-02",
"2006-01-02T15:04:05Z",
"2006-01-02T15:04:05-07:00",
"20060102",
"02-Jan-2006",
}
// Clean up value
value = strings.TrimSpace(value)
// Handle "YYYYMMDD" format from LACNIC
if len(value) == lacnicDateFormatLen {
t, err := time.Parse("20060102", value)
if err == nil {
return &t
}
}
for _, format := range formats {
t, err := time.Parse(format, value)
if err == nil {
return &t
}
}
return nil
}
// extractAbuseEmail extracts abuse email from response using regex.
func (c *Client) extractAbuseEmail(response string) string {
// Look for "Abuse contact for 'AS...' is 'email@domain'"
re := regexp.MustCompile(`[Aa]buse contact.*?is\s+['"]?([^\s'"]+@[^\s'"]+)['"]?`)
if matches := re.FindStringSubmatch(response); len(matches) > 1 {
return matches[1]
}
return ""
}
pkg/asinfo/asinfo.go
+4 -2
View File
@@ -1,4 +1,3 @@
// Package asinfo provides information about Autonomous Systems (AS)
package asinfo
import (
@@ -54,7 +53,9 @@ func initRegistry() {
}
var asInfos []Info
if err := json.Unmarshal(jsonData, &asInfos); err != nil {
err = json.Unmarshal(jsonData, &asInfos)
if err != nil {
panic("failed to unmarshal embedded AS data: " + err.Error())
}
@@ -137,6 +138,7 @@ func Search(query string) []Info {
defer defaultRegistry.mu.RUnlock()
var results []Info
for _, info := range defaultRegistry.byASN {
if contains(info.Handle, query) || contains(info.Description, query) {
results = append(results, *info)
pkg/asinfo/asinfo_test.go
+52 -16
View File
@@ -1,10 +1,15 @@
package asinfo
package asinfo_test
import (
"strings"
"testing"
"git.eeqj.de/sneak/routewatch/pkg/asinfo"
)
func TestGet(t *testing.T) {
t.Parallel()
tests := []struct {
name string
asn int
@@ -38,20 +43,26 @@ func TestGet(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
info, ok := Get(tt.asn)
t.Parallel()
info, ok := asinfo.Get(tt.asn)
if ok != tt.wantOK {
t.Errorf("Get(%d) ok = %v, want %v", tt.asn, ok, tt.wantOK)
return
}
if ok && info.Description != tt.wantDesc {
t.Errorf("Get(%d) description = %q, want %q", tt.asn, info.Description, tt.wantDesc)
t.Errorf("Get(%d) description = %q, want %q",
tt.asn, info.Description, tt.wantDesc)
}
})
}
}
func TestGetDescription(t *testing.T) {
t.Parallel()
tests := []struct {
asn int
want string
@@ -62,7 +73,10 @@ func TestGetDescription(t *testing.T) {
for _, tt := range tests {
t.Run("", func(t *testing.T) {
if got := GetDescription(tt.asn); got != tt.want {
t.Parallel()
got := asinfo.GetDescription(tt.asn)
if got != tt.want {
t.Errorf("GetDescription(%d) = %q, want %q", tt.asn, got, tt.want)
}
})
@@ -70,6 +84,8 @@ func TestGetDescription(t *testing.T) {
}
func TestGetHandle(t *testing.T) {
t.Parallel()
tests := []struct {
asn int
want string
@@ -81,7 +97,10 @@ func TestGetHandle(t *testing.T) {
for _, tt := range tests {
t.Run("", func(t *testing.T) {
if got := GetHandle(tt.asn); got != tt.want {
t.Parallel()
got := asinfo.GetHandle(tt.asn)
if got != tt.want {
t.Errorf("GetHandle(%d) = %q, want %q", tt.asn, got, tt.want)
}
})
@@ -89,18 +108,23 @@ func TestGetHandle(t *testing.T) {
}
func TestTotal(t *testing.T) {
total := Total()
t.Parallel()
total := asinfo.Total()
if total < 100000 {
t.Errorf("Total() = %d, expected > 100000", total)
}
// Verify it's consistent
if total2 := Total(); total2 != total {
total2 := asinfo.Total()
if total2 != total {
t.Errorf("Total() returned different values: %d vs %d", total, total2)
}
}
func TestSearch(t *testing.T) {
t.Parallel()
tests := []struct {
name string
query string
@@ -130,16 +154,21 @@ func TestSearch(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
results := Search(tt.query)
t.Parallel()
results := asinfo.Search(tt.query)
if len(results) < tt.wantMin {
t.Errorf("Search(%q) returned %d results, want at least %d", tt.query, len(results), tt.wantMin)
t.Errorf("Search(%q) returned %d results, want at least %d",
tt.query, len(results), tt.wantMin)
}
// Verify all results contain the query
if tt.query != "" {
for _, r := range results {
if !contains(r.Handle, tt.query) && !contains(r.Description, tt.query) {
t.Errorf("Search result ASN %d doesn't contain query %q", r.ASN, tt.query)
if !strings.Contains(r.Handle, tt.query) &&
!strings.Contains(r.Description, tt.query) {
t.Errorf("Search result ASN %d doesn't contain query %q",
r.ASN, tt.query)
}
}
}
@@ -148,13 +177,17 @@ func TestSearch(t *testing.T) {
}
func TestDataIntegrity(t *testing.T) {
t.Parallel()
// Verify no duplicate ASNs
all := All()
all := asinfo.All()
seen := make(map[int]bool)
for _, info := range all {
if seen[info.ASN] {
t.Errorf("Duplicate ASN found: %d", info.ASN)
}
seen[info.ASN] = true
}
@@ -163,6 +196,7 @@ func TestDataIntegrity(t *testing.T) {
if info.Handle == "" && info.ASN != 0 {
t.Errorf("ASN %d has empty handle", info.ASN)
}
if info.Description == "" {
t.Errorf("ASN %d has empty description", info.ASN)
}
@@ -174,8 +208,9 @@ func BenchmarkGet(b *testing.B) {
asns := []int{1, 15169, 13335, 32934, 8075, 16509}
b.ResetTimer()
for i := 0; i < b.N; i++ {
Get(asns[i%len(asns)])
for i := range b.N {
asinfo.Get(asns[i%len(asns)])
}
}
@@ -183,7 +218,8 @@ func BenchmarkSearch(b *testing.B) {
queries := []string{"Google", "Amazon", "Microsoft", "University"}
b.ResetTimer()
for i := 0; i < b.N; i++ {
Search(queries[i%len(queries)])
for i := range b.N {
asinfo.Search(queries[i%len(queries)])
}
}
script/bootstrap
Executable
+73
View File
@@ -0,0 +1,73 @@
#!/bin/sh
# script/bootstrap: install all dependencies needed to build and develop
# this repo. Idempotent: every install is guarded by a check so already
# installed tools are skipped. Base tooling comes from nix, apt, brew,
# or apk (detected in that order); assumes NOTHING is present (not git,
# make, or go). The linter and the tests run only in Docker, as phases of
# the Dockerfile, so nothing installs golangci-lint here.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
PKGMGR=""
SUDO=""
detect_pkgmgr() {
[ -n "$PKGMGR" ] && return 0
if command -v nix-env >/dev/null 2>&1; then
PKGMGR="nix"
elif command -v apt-get >/dev/null 2>&1; then
PKGMGR="apt"
elif command -v brew >/dev/null 2>&1; then
PKGMGR="brew"
elif command -v apk >/dev/null 2>&1; then
PKGMGR="apk"
else
echo "bootstrap: no supported package manager (nix, apt, brew, apk)" >&2
exit 1
fi
if [ "$PKGMGR" = "apt" ]; then
export DEBIAN_FRONTEND=noninteractive
if [ "$(id -u)" != "0" ]; then
SUDO="sudo"
fi
fi
}
# pkg_install <nix-attr> <apt-pkg> <brew-formula> <apk-pkg>
pkg_install() {
detect_pkgmgr
case "$PKGMGR" in
nix) nix-env -iA "nixpkgs.$1" ;;
apt) $SUDO env DEBIAN_FRONTEND=noninteractive apt-get install -y "$2" ;;
brew) brew install "$3" ;;
apk) apk add --no-cache "$4" ;;
esac
}
missing() {
! command -v "$1" >/dev/null 2>&1
}
main() {
cd "$ROOT"
# Base tooling (every repo)
if missing git; then pkg_install git git git git; fi
if missing make; then pkg_install gnumake make make make; fi
# Go toolchain
if missing go; then pkg_install go golang go go; fi
# Warn, don't fail: make build works without it.
if missing docker; then
echo "bootstrap: WARNING: docker not found; install it to run" \
"make test, make lint, make check and make docker." >&2
fi
go mod download
echo "bootstrap complete"
}
main "$@"
script/check
Executable
+16
View File
@@ -0,0 +1,16 @@
#!/bin/sh
# script/check: run all checks (test, lint, fmt-check). Our own
# extension to scripts-to-rule-them-all. test and lint are Docker
# phases; fmt-check is native, because a formatter writes the working
# tree. Must not modify any files.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
main() {
"$SCRIPT_DIR/test"
"$SCRIPT_DIR/lint"
"$SCRIPT_DIR/fmt-check"
}
main "$@"
script/cibuild
Executable
+28
View File
@@ -0,0 +1,28 @@
#!/bin/sh
# script/cibuild: run the CI build. It bootstraps first: a CI runner
# checks out and runs this and nothing else, and script/fmt-check runs
# the formatter on the host, which a pristine checkout cannot do.
# --no-cache for the same reason as script/docker: the gate phases the
# final stage depends on are RUN steps, and a cached one is a check that
# did not run.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
"$SCRIPT_DIR/bootstrap"
"$SCRIPT_DIR/check"
# Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an
# empty constant. The VERSION build argument takes precedence over
# the version a build stage derives from the .git in the context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$("$SCRIPT_DIR/projectname")" .
}
main "$@"
script/docker
Executable
+24
View File
@@ -0,0 +1,24 @@
#!/bin/sh
# script/docker: build the Docker image tagged with the project name.
# Identical in all repos; the tag comes from script/projectname.
# --no-cache because the gate phases the final stage depends on are RUN
# steps, and a cached one is a check that did not run.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
# Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an
# empty constant. The VERSION build argument takes precedence over
# the version a build stage derives from the .git in the context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$("$SCRIPT_DIR/projectname")" .
}
main "$@"
script/fmt
Executable
+12
View File
@@ -0,0 +1,12 @@
#!/bin/sh
# script/fmt: format all files (writes).
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
go fmt ./...
}
main "$@"
script/fmt-check
Executable
+18
View File
@@ -0,0 +1,18 @@
#!/bin/sh
# script/fmt-check: check formatting (read-only). Same scope as
# script/fmt, but fails instead of writing.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
unformatted="$(gofmt -l .)"
if [ -n "$unformatted" ]; then
echo "Files not formatted:" >&2
echo "$unformatted" >&2
exit 1
fi
}
main "$@"
script/install-precommit
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/bin/sh
# script/install-precommit: install the git pre-commit hook that runs
# script/precommit. Our own extension to scripts-to-rule-them-all.
# Generic: needs no adaptation.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
hook=".git/hooks/pre-commit"
printf '#!/bin/sh\nset -e\nscript/precommit\n' > .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit
echo "pre-commit hook installed: runs script/precommit"
}
main "$@"
script/lint
Executable
+23
View File
@@ -0,0 +1,23 @@
#!/bin/sh
# script/lint: run the linter. Linting is a phase of the Dockerfile and
# this builds that phase alone; the linter is never installed or run on
# a developer host, where a shared result cache and a host-global lock
# make its answer untrustworthy.
#
# The phase is not the last stage in the file, so it is built only when
# --target names it. --no-cache because a cached lint layer is a lint
# that did not run. The tag makes each build replace the previous image
# instead of leaving a dangling one behind.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
docker build --no-cache \
--target lint \
-t "$("$SCRIPT_DIR/projectname")-lint" .
}
main "$@"
script/precommit
Executable
+21
View File
@@ -0,0 +1,21 @@
#!/bin/sh
# script/precommit: run by the git pre-commit hook; fails the commit if
# checks fail. Our own extension to scripts-to-rule-them-all. Go repo
# extras: go mod tidy and go fmt must leave the tree unchanged.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
go mod tidy
go fmt ./...
git diff --exit-code -- go.mod go.sum || {
echo "go mod tidy changed files; please stage and retry"
exit 1
}
"$SCRIPT_DIR/check"
}
main "$@"
script/projectname
Executable
+12
View File
@@ -0,0 +1,12 @@
#!/bin/sh
# script/projectname: output the name of this project. Our own
# extension to scripts-to-rule-them-all. Other scripts that need the
# name (e.g. script/docker) call this, so they can stay identical
# across all repos.
set -eu
main() {
echo "routewatch"
}
main "$@"
script/setup
Executable
+14
View File
@@ -0,0 +1,14 @@
#!/bin/sh
# script/setup: set up the repo for development after a fresh clone:
# installs dependencies (script/bootstrap) and the git pre-commit hook.
# Add any repo-specific initialization (db init, .env template) here.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
main() {
"$SCRIPT_DIR/bootstrap"
"$SCRIPT_DIR/install-precommit"
}
main "$@"
script/test
Executable
+19
View File
@@ -0,0 +1,19 @@
#!/bin/sh
# script/test: run the test suite. Testing is a phase of the Dockerfile
# and this builds that phase alone, on the same terms as script/lint:
# --target because a phase that is not the last stage is built only when
# named, --no-cache because a cached test layer is a test that did not
# run, and a tag so each build replaces the previous image.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
docker build --no-cache \
--target test \
-t "$("$SCRIPT_DIR/projectname")-test" .
}
main "$@"