sneak
539d15c0a5
check / check (push) Waiting to run
The prefix distribution was the last query on the stats path. It read one index entry per live route on every request, so on a large database it took the whole 4-second deadline and /api/v1/stats answered 500. The distribution now lives next to the in-memory counts: seeded once at startup from the same query, then kept exact by the live-route writes. A new route whose prefix had no live route adds one at its mask length; a delete that leaves a prefix with no live route takes one away. Each check is one lookup on the prefix index, made only for new and removed routes, inside the write. Model: opus-5-5
108 lines
5.4 KiB
Markdown
108 lines
5.4 KiB
Markdown
# Workflow
|
|
|
|
* branch (from `main`)
|
|
* do the work in Next Step
|
|
* move Next Step to the top of Completed Steps
|
|
* move the top item of Future Steps into Next Step
|
|
* commit (`TODO.md` changes in the same commit as the work)
|
|
* merge to `main` if the branch is not protected, otherwise open a PR
|
|
* push
|
|
|
|
# Status
|
|
|
|
pre-1.0. No git tags. The Docker build runs the format check, the linter
|
|
and the tests, and the Gitea workflow runs that build on every push. The
|
|
image sets memory ceilings for a 5 GiB container (README "Memory") and the
|
|
README says how to run it under upaas (README "Running under upaas"). A
|
|
35-hour run of `3898daa` on the live feed peaked at about 1 GiB, without a
|
|
container memory limit.
|
|
|
|
# Next Step
|
|
|
|
`next` waits for sneak to merge it to `main` through
|
|
https://git.eeqj.de/sneak/routewatch/pulls/6. After that, setting
|
|
routewatch up under upaas on fsn1app1 and deploying it are his
|
|
(https://git.eeqj.de/sneak/routewatch/issues/31), and so is the run under a
|
|
real 5 GiB limit (https://git.eeqj.de/sneak/routewatch/issues/3).
|
|
|
|
# Completed Steps
|
|
|
|
- 2026-09-29: `/api/v1/stats` serves the prefix distribution from memory,
|
|
seeded at startup and adjusted on every live-route write, so a request no
|
|
longer reads every live route (closes #30)
|
|
- 2026-09-29: the entrypoint creates the data directory if it is missing and
|
|
stops the start if a step fails; README "Running under upaas" no longer
|
|
asks for the host directory to be created first (closes #42)
|
|
- 2026-09-29: `.dockerignore` keeps `.git`, local build output, local
|
|
databases and `.env` out of the Docker build context, and so out of the
|
|
source archive in the image (closes #39)
|
|
- 2026-09-29: README first line names the MIT license and the author;
|
|
the License section now says MIT and links `LICENSE`, and an Author
|
|
section was added; this file brought up to date (closes #38)
|
|
- 2026-09-29: MIT `LICENSE` (closes #1)
|
|
- 2026-09-28: stopping the daemon while the feed is flowing no longer
|
|
panics with "send on closed channel": the read loop checks for a stop
|
|
just before handing a message to the handler queues, and a second
|
|
`Stop` no longer closes the queues again (closes #34)
|
|
- 2026-09-28: `docker stop` no longer kills the daemon 2 seconds after the
|
|
stop signal: the entrypoint switches to the `routewatch` user with
|
|
`setpriv` instead of `runuser`, so the daemon receives the signal itself
|
|
and gets the whole wait `docker stop` allows, up to its own 60-second
|
|
limit (closes #33)
|
|
- 2026-09-28: ready to run under upaas: a set but invalid `PORT`,
|
|
`XDG_DATA_HOME` or `MALLOC_ARENA_MAX` stops the start, the health
|
|
check follows `PORT`, README "Running under upaas" section (closes
|
|
#31)
|
|
- 2026-09-22: realtime in-memory database statistics: counts seeded at
|
|
startup and adjusted on every write, oldest/newest route timestamps via
|
|
index-end lookups; `/api/v1/stats` no longer scans the tables (closes
|
|
#27)
|
|
- 2026-09-21: batch writes take the write lock when their transaction
|
|
begins (`_txlock=immediate`), so they wait out a WAL checkpoint instead
|
|
of failing with "database is locked" (closes #25)
|
|
- 2026-09-21: `MALLOC_ARENA_MAX=2` in the image caps glibc malloc arenas,
|
|
so memory outside the Go runtime no longer grows with the core count
|
|
(closes #23)
|
|
- 2026-09-21: `GOMEMLIMIT=1536MiB` in the image; README Memory section
|
|
with the memory budget and the 5 GiB container limit (closes #13)
|
|
- 2026-09-21: the four handler queues hold at most 20,000 messages each,
|
|
down from 100,000 (closes #11)
|
|
- 2026-09-21: two goroutine leaks fixed: the stats handlers after a
|
|
timeout and the streamer's tickers on every reconnect (closes #12)
|
|
- 2026-09-21: parsed RIS messages no longer keep the unused `Community`
|
|
and `Raw` fields (closes #9)
|
|
- 2026-09-21: `.editorconfig`, and a Gitea workflow that runs
|
|
`script/cibuild` on every push (closes #14)
|
|
- 2026-09-21: the peering handler's AS-path map holds at most 500,000
|
|
paths and is swapped for an empty one every 30 seconds instead of copied
|
|
(closes #10)
|
|
- 2026-09-21: SQLite memory bounded across the whole connection pool: a
|
|
64 MiB page cache on each connection, 1 GiB soft and 1.5 GiB hard heap
|
|
limits (closes #8)
|
|
- 2026-09-21: the Docker build runs the format check, the linter and the
|
|
tests, linting in a separate stage on a golangci-lint image pinned by
|
|
digest (closes #5)
|
|
- 2026-09-21: `make test` skips the live-network feed test (`-short`), so
|
|
`make check` no longer depends on the network (closes #2)
|
|
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints,
|
|
Makefile shims, README Entrypoints section
|
|
- 2026-02-22: repo policy compliance: required policy files, .gitignore
|
|
update, Makefile fmt-check/check/docker/hooks targets, gofmt pass
|
|
(repo-policies-compliance, unmerged)
|
|
- 2026-01-01: WAL checkpointing: periodic (5s), on startup with logging,
|
|
TRUNCATE mode; fixed slow queries
|
|
- 2025-12-31: status page: navbar, home page with search, oldest/newest
|
|
route timestamps, NULL handling in WHOIS stats; container runs as
|
|
routewatch user with proper state dir
|
|
- 2025-12-30: structured HTTP request logging, increased timeouts,
|
|
CIDR prefix URL routing fixes, status page metrics and footer
|
|
- 2025-12-29: Dockerfile multi-stage build with source archive; SQLite
|
|
incremental vacuum for non-blocking space reclamation
|
|
- 2025-07-27: initial RouteWatch BGP stream monitor
|
|
|
|
# Future Steps
|
|
|
|
- Production memory under 5 GiB: whether to test under a real 5 GiB
|
|
container limit on fsn1app1 is open for sneak
|
|
(https://git.eeqj.de/sneak/routewatch/issues/3)
|