The WHOIS fetcher's success and error time lists grow without bound unless the stats endpoints are polled #118

Closed
opened 2026-10-19 22:35:29 +02:00 by clawbot · 1 comment
Collaborator

Where: ASNFetcher.recordSuccess, recordFailure and GetStats in internal/routewatch/asnfetcher.go.

recordSuccess and recordFailure append time.Now() to successTimes and errorTimes on every fetch. The only pruning is in GetStats, which runs only when /status.json or /api/v1/stats is requested (the health check does not call it). A daemon nobody polls never prunes: during the initial fill of roughly 100,000 ASes (one fetch per second plus lookup time) the lists gain tens of thousands of entries a day, and about one entry per AS every 30 days after that, plus one per failure; a few MB a month, tens of MB a year, freed only by the first stats request, which then also copies the whole list.

Small against the 1.5 GiB Go budget, but memory that only a client request frees is the kind of growth #3 set out to remove.

Severity: low.

Definition of done

  • The lists are bounded without a stats request: prune entries older than statsWindow on append, or keep per-minute counters in a fixed ring.
  • Test, failing before the fix: call recordSuccess 100,000 times with an injected clock advancing past statsWindow without calling GetStats; the stored entries stay bounded by the window (today 100,000).
  • make check passes.

Model: fable-5-1

Where: `ASNFetcher.recordSuccess`, `recordFailure` and `GetStats` in `internal/routewatch/asnfetcher.go`. `recordSuccess` and `recordFailure` append `time.Now()` to `successTimes` and `errorTimes` on every fetch. The only pruning is in `GetStats`, which runs only when `/status.json` or `/api/v1/stats` is requested (the health check does not call it). A daemon nobody polls never prunes: during the initial fill of roughly 100,000 ASes (one fetch per second plus lookup time) the lists gain tens of thousands of entries a day, and about one entry per AS every 30 days after that, plus one per failure; a few MB a month, tens of MB a year, freed only by the first stats request, which then also copies the whole list. Small against the 1.5 GiB Go budget, but memory that only a client request frees is the kind of growth https://git.eeqj.de/sneak/routewatch/issues/3 set out to remove. Severity: low. ## Definition of done - The lists are bounded without a stats request: prune entries older than `statsWindow` on append, or keep per-minute counters in a fixed ring. - Test, failing before the fix: call `recordSuccess` 100,000 times with an injected clock advancing past `statsWindow` without calling `GetStats`; the stored entries stay bounded by the window (today 100,000). - `make check` passes. Model: fable-5-1
clawbot self-assigned this 2026-10-19 22:35:29 +02:00
Author
Collaborator

#143 drops the times older than statsWindow each time a success or failure is recorded, so both lists stay within an hour of fetches without a stats request, with the test this issue names.

Model: opus-5-5

https://git.eeqj.de/sneak/routewatch/pulls/143 drops the times older than `statsWindow` each time a success or failure is recorded, so both lists stay within an hour of fetches without a stats request, with the test this issue names. Model: opus-5-5
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/routewatch#118