sneak
9ca9c3c9fa
check / check (push) Canceled after 0s
README "HTTP Endpoints" lists every route setupRoutes registers with what it answers, and "Database Schema" matches schema.sql. Other sentences the code contradicts are corrected: the handlers' batch sizes, the stream client, the macOS state directory, the build command and what script/precommit checks; "Code Structure" gains the version and whois packages. TODO.md gets one line for each issue landed on next since 2026-10-03, and a current Status, Next Step and Future Steps. Model: opus-5-5
180 lines
10 KiB
Markdown
180 lines
10 KiB
Markdown
# Workflow
|
|
|
|
- branch (from `main`)
|
|
- do the work in Next Step
|
|
- move Next Step to the top of Completed Steps
|
|
- move the top item of Future Steps into Next Step
|
|
- commit (`TODO.md` changes in the same commit as the work)
|
|
- merge to `main` if the branch is not protected, otherwise open a PR
|
|
- push
|
|
|
|
# Status
|
|
|
|
pre-1.0. No git tags. `main` last took `next` on 2026-10-06, through
|
|
https://git.eeqj.de/sneak/routewatch/pulls/6. The linter and the tests are
|
|
phases of the Docker build, and the Gitea workflow runs `script/cibuild` (the
|
|
format check of the Go code and the markdown, both phases, then the image) on
|
|
every push. The image sets memory ceilings for a 5 GiB container (README
|
|
"Memory") and the README says how to run it under upaas (README "Running under
|
|
upaas"). A 25-hour run of `a9b6860` on the live feed peaked at about 0.6 GiB
|
|
(https://git.eeqj.de/sneak/routewatch/issues/3). A run of `825afa2` on the live
|
|
feed found the prefix handler dropping about a quarter of its updates because
|
|
its route writes were slower than the feed
|
|
(https://git.eeqj.de/sneak/routewatch/issues/96); the cheaper writes since then
|
|
are measured by a benchmark, not yet on the live feed.
|
|
|
|
# Next Step
|
|
|
|
`next` waits for sneak to merge it to `main` through
|
|
https://git.eeqj.de/sneak/routewatch/pulls/87.
|
|
|
|
# Completed Steps
|
|
|
|
- 2026-10-19: README "HTTP Endpoints" lists every route with what it answers,
|
|
"Database Schema" matches `schema.sql`, and the rest of the README matches the
|
|
code; this file brought up to date (closes #72)
|
|
- 2026-10-19: route writes are cheaper: the live route and prefix tables lose
|
|
their random UUID keys and the prefix tables their duplicate index, a route
|
|
update sets only the columns that change, and every write goes through one
|
|
connection with a page cache of 1 GiB (README "Memory"). A benchmark writes
|
|
the prefix handler's largest batch faster than the feed sends it (closes #96)
|
|
- 2026-10-19: the WAL checkpoint, the incremental vacuum and ANALYZE take the
|
|
database lock every batch write holds, so a vacuum no longer fails with
|
|
"database is locked" and an ANALYZE no longer makes a batch write lose its
|
|
updates; ANALYZE runs as `PRAGMA optimize` with an analysis limit (closes #94)
|
|
- 2026-10-19: shutdown stops the streamer first, and each handler writes
|
|
everything left on its queue before it stops, so the updates waiting there are
|
|
no longer lost (closes #69)
|
|
- 2026-10-19: a path that ends in an AS set of several members has origin AS 0,
|
|
no known origin, instead of the set's last member, and the set adds no
|
|
peerings; the pages show such a route as having no known origin (closes #75)
|
|
- 2026-10-19: `bgp_peer_count` drops to 0 when the last peer goes down (closes
|
|
#92)
|
|
- 2026-10-19: `bgp_peer_count` counts every peer the feed has heard from, not
|
|
only those whose session opened since startup, and drops a peer whose session
|
|
goes down (closes #74)
|
|
- 2026-10-19: the shared files are re-vendored from `sneak/prompts` at
|
|
`0653579`: golangci-lint v2.14.0 and its configuration, lint and test as
|
|
`Dockerfile` phases that keep Go's and golangci-lint's caches, markdown
|
|
formatted with prettier, agent guidance in `AGENTS.md`; the code is fixed for
|
|
what the new linter configuration finds (closes #52)
|
|
- 2026-10-19: the streamer reads feed lines up to 1 MiB, so a line over 64 KiB
|
|
no longer ends the connection (closes #76)
|
|
- 2026-10-19: a peer's live routes are removed when its BGP session goes down;
|
|
`STATE` messages are never dropped for backpressure, and one that finds its
|
|
queue full waits for room ahead of the handler's other messages (closes #56)
|
|
- 2026-10-19: the prefix-length pages pick their sample with a bounded number of
|
|
index lookups instead of reading every route of that length, and list each
|
|
prefix once (closes #66)
|
|
- 2026-10-19: a request past its time limit answers 504 instead of 408, and
|
|
`/api/v1/stats` answers 504 instead of 500 when its read runs out of time
|
|
(closes #70)
|
|
- 2026-10-19: the AS, peer and prefix handlers write each batch outside the lock
|
|
their message handling needs, so messages keep coming off their queues during
|
|
a write; meanwhile the next batch grows to at most ten batch sizes (closes
|
|
#65)
|
|
- 2026-10-19: a live route is identified by prefix and peer, so an announcement
|
|
with a new origin AS replaces the peer's route and a withdrawal removes it
|
|
whatever its origin AS (closes #55)
|
|
- 2026-10-14: the streamer reconnects when RIS Live ends the stream or sends
|
|
nothing for 60 seconds, and the health check reports the feed disconnected
|
|
meanwhile (closes #53)
|
|
- 2026-10-14: the prefix page takes the names of the origin ASes from the prefix
|
|
query instead of reading every route of each origin AS (closes #67)
|
|
- 2026-10-14: the daemon exits with status 1 when its web server cannot listen
|
|
on its port, instead of running on without it (closes #73)
|
|
- 2026-10-14: the prefix pages parse the prefix before looking it up: one that
|
|
does not parse answers 400, and any valid way of writing a prefix finds its
|
|
routes (closes #71)
|
|
- 2026-10-14: the IP lookup ignores routes shorter than /8 for IPv4 and /16 for
|
|
IPv6, so a default route is no longer named as an address's owner (closes #57)
|
|
- 2026-10-14: a prefix batch keeps each peer's last update for a prefix, instead
|
|
of one update per prefix across all peers (closes #54)
|
|
- 2026-10-14: `reconnect_count` in `/api/v1/stats` counts every reconnect
|
|
(closes #68)
|
|
- 2026-10-03: `/api/v1/stats` serves the prefix distribution from memory, seeded
|
|
at startup and adjusted on every live-route write, so a request no longer
|
|
reads every live route (closes #30)
|
|
- 2026-10-03: a new database is created with `auto_vacuum` set to incremental,
|
|
through the connection string so it is set before the file is first written,
|
|
and each periodic incremental vacuum now returns up to 1000 free pages; the
|
|
late `PRAGMA auto_vacuum` in `Initialize`, which SQLite ignored, is gone
|
|
(closes #43)
|
|
- 2026-10-03: looking up an IP address no longer reads every IPv6 route: both
|
|
families find the most specific live route with at most 33 or 129 lookups on
|
|
the prefix index, and the IPv4 range columns are gone. Prefixes from the feed
|
|
are stored in one text form, so an IPv6 withdrawal, which the feed sends
|
|
uncompressed, now removes its route (closes #48)
|
|
- 2026-10-02: a plain `docker build .` stamps the commit's tag or short commit
|
|
(`git describe --tags --always`) into the page footer instead of `unknown`:
|
|
`.dockerignore` sends `.git` without `.git/config`, a `VERSION` build argument
|
|
takes precedence, and `make build` stamps the same value (closes #46)
|
|
- 2026-09-29: the entrypoint creates the data directory if it is missing and
|
|
stops the start if a step fails; README "Running under upaas" no longer asks
|
|
for the host directory to be created first (closes #42)
|
|
- 2026-09-29: `.dockerignore` keeps `.git`, local build output, local databases
|
|
and `.env` out of the Docker build context, and so out of the source archive
|
|
in the image (closes #39)
|
|
- 2026-09-29: README first line names the MIT license and the author; the
|
|
License section now says MIT and links `LICENSE`, and an Author section was
|
|
added; this file brought up to date (closes #38)
|
|
- 2026-09-29: MIT `LICENSE` (closes #1)
|
|
- 2026-09-28: stopping the daemon while the feed is flowing no longer panics
|
|
with "send on closed channel": the read loop checks for a stop just before
|
|
handing a message to the handler queues, and a second `Stop` no longer closes
|
|
the queues again (closes #34)
|
|
- 2026-09-28: `docker stop` no longer kills the daemon 2 seconds after the stop
|
|
signal: the entrypoint switches to the `routewatch` user with `setpriv`
|
|
instead of `runuser`, so the daemon receives the signal itself and gets the
|
|
whole wait `docker stop` allows, up to its own 60-second limit (closes #33)
|
|
- 2026-09-28: ready to run under upaas: a set but invalid `PORT`,
|
|
`XDG_DATA_HOME` or `MALLOC_ARENA_MAX` stops the start, the health check
|
|
follows `PORT`, README "Running under upaas" section (closes #31)
|
|
- 2026-09-22: realtime in-memory database statistics: counts seeded at startup
|
|
and adjusted on every write, oldest/newest route timestamps via index-end
|
|
lookups; `/api/v1/stats` no longer scans the tables (closes #27)
|
|
- 2026-09-21: batch writes take the write lock when their transaction begins
|
|
(`_txlock=immediate`), so they wait out a WAL checkpoint instead of failing
|
|
with "database is locked" (closes #25)
|
|
- 2026-09-21: `MALLOC_ARENA_MAX=2` in the image caps glibc malloc arenas, so
|
|
memory outside the Go runtime no longer grows with the core count (closes #23)
|
|
- 2026-09-21: `GOMEMLIMIT=1536MiB` in the image; README Memory section with the
|
|
memory budget and the 5 GiB container limit (closes #13)
|
|
- 2026-09-21: the four handler queues hold at most 20,000 messages each, down
|
|
from 100,000 (closes #11)
|
|
- 2026-09-21: two goroutine leaks fixed: the stats handlers after a timeout and
|
|
the streamer's tickers on every reconnect (closes #12)
|
|
- 2026-09-21: parsed RIS messages no longer keep the unused `Community` and
|
|
`Raw` fields (closes #9)
|
|
- 2026-09-21: `.editorconfig`, and a Gitea workflow that runs `script/cibuild`
|
|
on every push (closes #14)
|
|
- 2026-09-21: the peering handler's AS-path map holds at most 500,000 paths and
|
|
is swapped for an empty one every 30 seconds instead of copied (closes #10)
|
|
- 2026-09-21: SQLite memory bounded across the whole connection pool: a 64 MiB
|
|
page cache on each connection, 1 GiB soft and 1.5 GiB hard heap limits (closes
|
|
#8)
|
|
- 2026-09-21: the Docker build runs the format check, the linter and the tests,
|
|
linting in a separate stage on a golangci-lint image pinned by digest (closes
|
|
#5)
|
|
- 2026-09-21: `make test` skips the live-network feed test (`-short`), so
|
|
`make check` no longer depends on the network (closes #2)
|
|
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile
|
|
shims, README Entrypoints section
|
|
- 2026-02-22: repo policy compliance: required policy files, .gitignore update,
|
|
Makefile fmt-check/check/docker/hooks targets, gofmt pass
|
|
(repo-policies-compliance, unmerged)
|
|
- 2026-01-01: WAL checkpointing: periodic (5s), on startup with logging,
|
|
TRUNCATE mode; fixed slow queries
|
|
- 2025-12-31: status page: navbar, home page with search, oldest/newest route
|
|
timestamps, NULL handling in WHOIS stats; container runs as routewatch user
|
|
with proper state dir
|
|
- 2025-12-30: structured HTTP request logging, increased timeouts, CIDR prefix
|
|
URL routing fixes, status page metrics and footer
|
|
- 2025-12-29: Dockerfile multi-stage build with source archive; SQLite
|
|
incremental vacuum for non-blocking space reclamation
|
|
- 2025-07-27: initial RouteWatch BGP stream monitor
|
|
|
|
# Future Steps
|
|
|
|
None open.
|