routewatch/TODO.md

# Workflow

* branch (from `main`)
* do the work in Next Step
* move Next Step to the top of Completed Steps
* move the top item of Future Steps into Next Step
* commit (`TODO.md` changes in the same commit as the work)
* merge to `main` if the branch is not protected, otherwise open a PR
* push

# Status

pre-1.0. No git tags. The Docker build runs the format check, the linter
and the tests, and the Gitea workflow runs that build on every push. The
image sets memory ceilings for a 5 GiB container (README "Memory") and the
README says how to run it under upaas (README "Running under upaas"). A
35-hour run of `3898daa` on the live feed peaked at about 1 GiB, without a
container memory limit.

# Next Step

`next` waits for sneak to merge it to `main` through
https://git.eeqj.de/sneak/routewatch/pulls/6. After that, setting
routewatch up under upaas on fsn1app1 and deploying it are his
(https://git.eeqj.de/sneak/routewatch/issues/31), and so is the run under a
real 5 GiB limit (https://git.eeqj.de/sneak/routewatch/issues/3).
The other open issue is https://git.eeqj.de/sneak/routewatch/issues/30.

# Completed Steps

- 2026-09-29: the entrypoint creates the data directory if it is missing and
  stops the start if a step fails; README "Running under upaas" no longer
  asks for the host directory to be created first (closes #42)
- 2026-09-29: `.dockerignore` keeps `.git`, local build output, local
  databases and `.env` out of the Docker build context, and so out of the
  source archive in the image (closes #39)
- 2026-09-29: README first line names the MIT license and the author;
  the License section now says MIT and links `LICENSE`, and an Author
  section was added; this file brought up to date (closes #38)
- 2026-09-29: MIT `LICENSE` (closes #1)
- 2026-09-28: stopping the daemon while the feed is flowing no longer
  panics with "send on closed channel": the read loop checks for a stop
  just before handing a message to the handler queues, and a second
  `Stop` no longer closes the queues again (closes #34)
- 2026-09-28: `docker stop` no longer kills the daemon 2 seconds after the
  stop signal: the entrypoint switches to the `routewatch` user with
  `setpriv` instead of `runuser`, so the daemon receives the signal itself
  and gets the whole wait `docker stop` allows, up to its own 60-second
  limit (closes #33)
- 2026-09-28: ready to run under upaas: a set but invalid `PORT`,
  `XDG_DATA_HOME` or `MALLOC_ARENA_MAX` stops the start, the health
  check follows `PORT`, README "Running under upaas" section (closes
  #31)
- 2026-09-22: realtime in-memory database statistics: counts seeded at
  startup and adjusted on every write, oldest/newest route timestamps via
  index-end lookups; `/api/v1/stats` no longer scans the tables (closes
  #27)
- 2026-09-21: batch writes take the write lock when their transaction
  begins (`_txlock=immediate`), so they wait out a WAL checkpoint instead
  of failing with "database is locked" (closes #25)
- 2026-09-21: `MALLOC_ARENA_MAX=2` in the image caps glibc malloc arenas,
  so memory outside the Go runtime no longer grows with the core count
  (closes #23)
- 2026-09-21: `GOMEMLIMIT=1536MiB` in the image; README Memory section
  with the memory budget and the 5 GiB container limit (closes #13)
- 2026-09-21: the four handler queues hold at most 20,000 messages each,
  down from 100,000 (closes #11)
- 2026-09-21: two goroutine leaks fixed: the stats handlers after a
  timeout and the streamer's tickers on every reconnect (closes #12)
- 2026-09-21: parsed RIS messages no longer keep the unused `Community`
  and `Raw` fields (closes #9)
- 2026-09-21: `.editorconfig`, and a Gitea workflow that runs
  `script/cibuild` on every push (closes #14)
- 2026-09-21: the peering handler's AS-path map holds at most 500,000
  paths and is swapped for an empty one every 30 seconds instead of copied
  (closes #10)
- 2026-09-21: SQLite memory bounded across the whole connection pool: a
  64 MiB page cache on each connection, 1 GiB soft and 1.5 GiB hard heap
  limits (closes #8)
- 2026-09-21: the Docker build runs the format check, the linter and the
  tests, linting in a separate stage on a golangci-lint image pinned by
  digest (closes #5)
- 2026-09-21: `make test` skips the live-network feed test (`-short`), so
  `make check` no longer depends on the network (closes #2)
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints,
  Makefile shims, README Entrypoints section
- 2026-02-22: repo policy compliance: required policy files, .gitignore
  update, Makefile fmt-check/check/docker/hooks targets, gofmt pass
  (repo-policies-compliance, unmerged)
- 2026-01-01: WAL checkpointing: periodic (5s), on startup with logging,
  TRUNCATE mode; fixed slow queries
- 2025-12-31: status page: navbar, home page with search, oldest/newest
  route timestamps, NULL handling in WHOIS stats; container runs as
  routewatch user with proper state dir
- 2025-12-30: structured HTTP request logging, increased timeouts,
  CIDR prefix URL routing fixes, status page metrics and footer
- 2025-12-29: Dockerfile multi-stage build with source archive; SQLite
  incremental vacuum for non-blocking space reclamation
- 2025-07-27: initial RouteWatch BGP stream monitor

# Future Steps

- Production memory under 5 GiB: whether to test under a real 5 GiB
  container limit on fsn1app1 is open for sneak
  (https://git.eeqj.de/sneak/routewatch/issues/3)
- `/api/v1/stats` answered HTTP 500 after 35 hours on the live feed, seen
  on `3898daa`, which predates the 2026-09-22 in-memory statistics
  (https://git.eeqj.de/sneak/routewatch/issues/30)