Next milestone: production memory under 5 GiB, ready for upaas #6

Open
clawbot wants to merge 19 commits from next into main
2 changed files with 61 additions and 14 deletions
Showing only changes of commit 87a3bc115e - Show all commits
+9 -2
View File
@@ -1,6 +1,9 @@
# RouteWatch # RouteWatch
RouteWatch is a real-time BGP routing table monitor that streams BGP UPDATE messages from the RIPE RIS Live service, maintains a live routing table in SQLite, and provides HTTP APIs for querying routing information. RouteWatch is an MIT-licensed Go daemon by @sneak that monitors the BGP routing
table in real time: it streams BGP UPDATE messages from the RIPE RIS Live
service, maintains a live routing table in SQLite, and provides HTTP APIs for
querying routing information.
## Features ## Features
@@ -299,4 +302,8 @@ invoke `go test` directly without `-short`:
## License ## License
See LICENSE file. MIT. See [`LICENSE`](LICENSE).
## Author
[@sneak](https://sneak.berlin)
+52 -12
View File
@@ -10,19 +10,30 @@
# Status # Status
pre-1.0. No git tags. Runs in production-style Docker deployment, but pre-1.0. No git tags. The Docker build runs the format check, the linter
the policy compliance branch (repo-policies-compliance, make check and the tests, and the Gitea workflow runs that build on every push. The
passing, clean tree) is unmerged to main and the CI workflow is missing. image sets memory ceilings for a 5 GiB container (README "Memory") and the
README says how to run it under upaas (README "Running under upaas"). A
35-hour run of `3898daa` on the live feed peaked at about 1 GiB, without a
container memory limit.
# Next Step # Next Step
Merge repo-policies-compliance into main (3 commits: policy files and `next` waits for sneak to merge it to `main` through
.gitignore, Makefile targets fmt-check/check/docker/hooks, gofmt pass), https://git.eeqj.de/sneak/routewatch/pulls/6. After that, setting
then add .gitea/workflows/check.yml as a small follow-up commit so CI routewatch up under upaas on fsn1app1 and deploying it are his
runs make check on main. (https://git.eeqj.de/sneak/routewatch/issues/31), and so is the run under a
real 5 GiB limit (https://git.eeqj.de/sneak/routewatch/issues/3).
Nothing on the record orders
https://git.eeqj.de/sneak/routewatch/issues/39 and
https://git.eeqj.de/sneak/routewatch/issues/30.
# Completed Steps # Completed Steps
- 2026-09-29: README first line names the MIT license and the author;
the License section now says MIT and links `LICENSE`, and an Author
section was added; this file brought up to date (closes #38)
- 2026-09-29: MIT `LICENSE` (closes #1)
- 2026-09-28: stopping the daemon while the feed is flowing no longer - 2026-09-28: stopping the daemon while the feed is flowing no longer
panics with "send on closed channel": the read loop checks for a stop panics with "send on closed channel": the read loop checks for a stop
just before handing a message to the handler queues, and a second just before handing a message to the handler queues, and a second
@@ -40,6 +51,33 @@ runs make check on main.
startup and adjusted on every write, oldest/newest route timestamps via startup and adjusted on every write, oldest/newest route timestamps via
index-end lookups; `/api/v1/stats` no longer scans the tables (closes index-end lookups; `/api/v1/stats` no longer scans the tables (closes
#27) #27)
- 2026-09-21: batch writes take the write lock when their transaction
begins (`_txlock=immediate`), so they wait out a WAL checkpoint instead
of failing with "database is locked" (closes #25)
- 2026-09-21: `MALLOC_ARENA_MAX=2` in the image caps glibc malloc arenas,
so memory outside the Go runtime no longer grows with the core count
(closes #23)
- 2026-09-21: `GOMEMLIMIT=1536MiB` in the image; README Memory section
with the memory budget and the 5 GiB container limit (closes #13)
- 2026-09-21: the four handler queues hold at most 20,000 messages each,
down from 100,000 (closes #11)
- 2026-09-21: two goroutine leaks fixed: the stats handlers after a
timeout and the streamer's tickers on every reconnect (closes #12)
- 2026-09-21: parsed RIS messages no longer keep the unused `Community`
and `Raw` fields (closes #9)
- 2026-09-21: `.editorconfig`, and a Gitea workflow that runs
`script/cibuild` on every push (closes #14)
- 2026-09-21: the peering handler's AS-path map holds at most 500,000
paths and is swapped for an empty one every 30 seconds instead of copied
(closes #10)
- 2026-09-21: SQLite memory bounded across the whole connection pool: a
64 MiB page cache on each connection, 1 GiB soft and 1.5 GiB hard heap
limits (closes #8)
- 2026-09-21: the Docker build runs the format check, the linter and the
tests, linting in a separate stage on a golangci-lint image pinned by
digest (closes #5)
- 2026-09-21: `make test` skips the live-network feed test (`-short`), so
`make check` no longer depends on the network (closes #2)
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, - 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints,
Makefile shims, README Entrypoints section Makefile shims, README Entrypoints section
- 2026-02-22: repo policy compliance: required policy files, .gitignore - 2026-02-22: repo policy compliance: required policy files, .gitignore
@@ -58,8 +96,10 @@ runs make check on main.
# Future Steps # Future Steps
- Verify main is green after the merge: make check locally and the new - Production memory under 5 GiB: whether to test under a real 5 GiB
CI workflow passing container limit on fsn1app1 is open for sneak
- Review stale remote branches fix-min-time-calculation and (https://git.eeqj.de/sneak/routewatch/issues/3)
optimize-sqlite-settings: land or delete - `/api/v1/stats` answered HTTP 500 after 35 hours on the live feed, seen
- Clean up the tmp/ directory at the repo root: gitignore or remove on `3898daa`, which predates the 2026-09-22 in-memory statistics
(https://git.eeqj.de/sneak/routewatch/issues/30)
- Add a `.dockerignore` (https://git.eeqj.de/sneak/routewatch/issues/39)