Once the database passed a few GiB, every /api/v1/stats request ran a COUNT(*) over each table plus a MIN/MAX union scan of both route tables, hit the request timeout and returned HTTP 500, so the status page went blank (#27).
Per the owner's ruling on that issue, this replaces the 30 s recompute from #28 with realtime maintenance. The row counts (ASNs, prefixes, peerings, peers, live routes v4/v6) are seeded once at startup and adjusted on every write as records are inserted, updated and removed, all under one mutex. A read serves them from memory: no scan, no timer, always current.
A live-route upsert is now an UPDATE followed by an INSERT only when no row matched, so the update's rows-affected count distinguishes a new route from a re-announced one and keeps the counters exact. Oldest/newest timestamps read one row from each end of the last_updated index (also fixing the per-call scan warning). JSON shape is unchanged.
Notes / disclosures
Beyond the issue's named files: internal/database/interface.go gains two Stats fields; new internal/database/counts.go and its test.
Supersedes #28 (its cache is dropped); the timestamp index-scan fix is kept.
Judgement call: the distinct-prefix histogram stays a single indexed query, not an in-memory counter — an in-memory form would need a per-prefix table of roughly a million entries, memory this service is tuned to avoid. Oldest/newest likewise use index-end lookups (log-time, not scans) rather than counters, since an exact in-memory min would mean indexing every route's timestamp.
script/cibuild is green.
Model: opus-4-8
## What and why
Once the database passed a few GiB, every `/api/v1/stats` request ran a `COUNT(*)` over each table plus a `MIN`/`MAX` union scan of both route tables, hit the request timeout and returned HTTP 500, so the status page went blank (https://git.eeqj.de/sneak/routewatch/issues/27).
Per the owner's ruling on that issue, this replaces the 30 s recompute from https://git.eeqj.de/sneak/routewatch/pulls/28 with realtime maintenance. The row counts (ASNs, prefixes, peerings, peers, live routes v4/v6) are seeded once at startup and adjusted on every write as records are inserted, updated and removed, all under one mutex. A read serves them from memory: no scan, no timer, always current.
A live-route upsert is now an `UPDATE` followed by an `INSERT` only when no row matched, so the update's rows-affected count distinguishes a new route from a re-announced one and keeps the counters exact. Oldest/newest timestamps read one row from each end of the `last_updated` index (also fixing the per-call scan warning). JSON shape is unchanged.
## Notes / disclosures
- Beyond the issue's named files: `internal/database/interface.go` gains two `Stats` fields; new `internal/database/counts.go` and its test.
- Supersedes https://git.eeqj.de/sneak/routewatch/pulls/28 (its cache is dropped); the timestamp index-scan fix is kept.
- Judgement call: the distinct-prefix histogram stays a single indexed query, not an in-memory counter — an in-memory form would need a per-prefix table of roughly a million entries, memory this service is tuned to avoid. Oldest/newest likewise use index-end lookups (log-time, not scans) rather than counters, since an exact in-memory min would mean indexing every route's timestamp.
- `script/cibuild` is green.
Model: opus-4-8
The stats endpoints recomputed every count with a COUNT(*) over each
table plus a MIN/MAX union scan on every request, which took the whole
timeout and returned 500 once the database passed a few GiB.
The row counts (ASNs, prefixes, peerings, peers, live routes) now live
in memory: seeded once at startup and adjusted on each write as routes
and records are inserted, updated and removed, so a read serves them
with no scan and no timer. A live-route upsert is an UPDATE, then an
INSERT only when no row matched, so its rows-affected count tells a new
route from a re-announced one. Oldest/newest timestamps read one row
from each end of the last_updated index, which also fixes the per-call
scan warning. The distinct-prefix histogram stays a single indexed
query.
JSON shape of /api/v1/stats and the status page is unchanged.
Model: opus-4-8
The /api/v1/stats counts are served from truly incremental in-memory counters — seeded once at startup and then adjusted on every insert, update and delete under a mutex, with no periodic recompute and no timer — which matches #27 and the owner's realtime ruling; every counted write path increments only on an actual insert (and decrements by exact rows-affected on withdrawal), the meaningful tests cover the incremental behaviour (re-announce holds the counts, withdrawal drops them), startup seeding and concurrent read/write, and the full test/lint/fmt-check suite passes in Docker on the head rebased onto current next.
Disclosure: the per-request oldest/newest lookups and the prefix-distribution query remain indexed reads rather than counters — a disclosed judgement call in the PR, consistent with the owner exchange.
Model: opus-4-8
**PASS**
The `/api/v1/stats` counts are served from truly incremental in-memory counters — seeded once at startup and then adjusted on every insert, update and delete under a mutex, with no periodic recompute and no timer — which matches https://git.eeqj.de/sneak/routewatch/issues/27 and the owner's realtime ruling; every counted write path increments only on an actual insert (and decrements by exact rows-affected on withdrawal), the meaningful tests cover the incremental behaviour (re-announce holds the counts, withdrawal drops them), startup seeding and concurrent read/write, and the full test/lint/fmt-check suite passes in Docker on the head rebased onto current `next`.
Disclosure: the per-request oldest/newest lookups and the prefix-distribution query remain indexed reads rather than counters — a disclosed judgement call in the PR, consistent with the owner exchange.
Model: opus-4-8
clawbot
merged commit df9e23d503 into next2026-09-22 09:41:16 +02:00
Squash-merged to next on the passed independent review above. Issue 27 closes when next merges to main. routewatch now idles per the 2026-09-22 priority ruling; the issue 3 memory-cap implementation stays queued for when the repo is reactivated.
model: claude-fable-5
Squash-merged to next on the passed independent review above. Issue 27 closes when next merges to main. routewatch now idles per the 2026-09-22 priority ruling; the issue 3 memory-cap implementation stays queued for when the repo is reactivated.
model: claude-fable-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What and why
Once the database passed a few GiB, every
/api/v1/statsrequest ran aCOUNT(*)over each table plus aMIN/MAXunion scan of both route tables, hit the request timeout and returned HTTP 500, so the status page went blank (#27).Per the owner's ruling on that issue, this replaces the 30 s recompute from #28 with realtime maintenance. The row counts (ASNs, prefixes, peerings, peers, live routes v4/v6) are seeded once at startup and adjusted on every write as records are inserted, updated and removed, all under one mutex. A read serves them from memory: no scan, no timer, always current.
A live-route upsert is now an
UPDATEfollowed by anINSERTonly when no row matched, so the update's rows-affected count distinguishes a new route from a re-announced one and keeps the counters exact. Oldest/newest timestamps read one row from each end of thelast_updatedindex (also fixing the per-call scan warning). JSON shape is unchanged.Notes / disclosures
internal/database/interface.gogains twoStatsfields; newinternal/database/counts.goand its test.script/cibuildis green.Model: opus-4-8
PASS
The
/api/v1/statscounts are served from truly incremental in-memory counters — seeded once at startup and then adjusted on every insert, update and delete under a mutex, with no periodic recompute and no timer — which matches #27 and the owner's realtime ruling; every counted write path increments only on an actual insert (and decrements by exact rows-affected on withdrawal), the meaningful tests cover the incremental behaviour (re-announce holds the counts, withdrawal drops them), startup seeding and concurrent read/write, and the full test/lint/fmt-check suite passes in Docker on the head rebased onto currentnext.Disclosure: the per-request oldest/newest lookups and the prefix-distribution query remain indexed reads rather than counters — a disclosed judgement call in the PR, consistent with the owner exchange.
Model: opus-4-8
Squash-merged to next on the passed independent review above. Issue 27 closes when next merges to main. routewatch now idles per the 2026-09-22 priority ruling; the issue 3 memory-cap implementation stays queued for when the repo is reactivated.
model: claude-fable-5