Reduce the four handler queues from 100,000 to 20,000 messages #11

New Issue

2026-09-21T14:54:36+02:00

clawbot commented

2026-09-21 14:54:36 +02:00

Part of #3 (analysis and plan in the comment dated 2026-09-21 14:34 there).

Each handler queue holds 100,000 message pointers (ashandler.go:15, peerhandler.go:18, prefixhandler.go:19, peeringhandler.go:17 in internal/routewatch); a message lives until the slowest handler drains it; derived worst case 800 MiB with all four full. The streamer already drops instead of blocking when a queue is full (internal/streamer/streamer.go:670-716), so a smaller queue is safe.

Requirements

Set all four constants to 20000 (about 4 s of feed). The largest batch size is asnBatchSize 30000: check how batches are filled; if a batch can only flush once the queue has delivered a full batch, state on the PR how flushing still happens (timer) and do not change batch sizes unless a test proves it necessary.
Update the comments next to the constants and any README text quoting 100,000.
Must be branched after the peering map issue has merged (same file).

Definition of done

make check green; any test that hard-codes the old capacity updated. Commit title ends (closes #N).

Model: fable-5-1

Part of https://git.eeqj.de/sneak/routewatch/issues/3 (analysis and plan in the comment dated 2026-09-21 14:34 there). Each handler queue holds 100,000 message pointers (`ashandler.go:15`, `peerhandler.go:18`, `prefixhandler.go:19`, `peeringhandler.go:17` in `internal/routewatch`); a message lives until the slowest handler drains it; derived worst case 800 MiB with all four full. The streamer already drops instead of blocking when a queue is full (`internal/streamer/streamer.go:670-716`), so a smaller queue is safe. ## Requirements - Set all four constants to 20000 (about 4 s of feed). The largest batch size is `asnBatchSize` 30000: check how batches are filled; if a batch can only flush once the queue has delivered a full batch, state on the PR how flushing still happens (timer) and do not change batch sizes unless a test proves it necessary. - Update the comments next to the constants and any README text quoting 100,000. - Must be branched after the peering map issue has merged (same file). ## Definition of done - `make check` green; any test that hard-codes the old capacity updated. Commit title ends ` (closes #N)`. Model: fable-5-1

clawbot referenced this issue

2026-09-21 14:55:52 +02:00

Production memory: run under 5 GiB on real internet feed (was killed at ~12 GiB) #3

clawbot referenced this issue from a commit

2026-09-21 16:09:45 +02:00

Shrink the four handler queues from 100,000 to 20,000 (closes #11)

clawbot referenced this issue

2026-09-21 16:10:01 +02:00

Shrink the four handler queues from 100,000 to 20,000 (closes #11) #20

clawbot referenced a pull request that will close this issue

2026-09-21 16:10:01 +02:00

Shrink the four handler queues from 100,000 to 20,000 (closes #11) #21

clawbot referenced this issue

2026-09-21 16:10:19 +02:00

Shrink the four handler queues from 100,000 to 20,000 (closes #11) #20

clawbot commented

2026-09-21 16:10:37 +02:00

Done in #21. All four handler queue capacities set to 20,000 with updated comments. Batch sizes left unchanged: asnBatchSize (30,000) now exceeds its queue, but each handler flushes on its own timer regardless of batch fill, and one queued UPDATE contributes every ASN in its path, so flushing is unaffected and no size change is needed. No README text quoted the old figure. TODO.md not touched, matching the other memory-reduction units.

Model: opus-4-8

Done in https://git.eeqj.de/sneak/routewatch/pulls/21. All four handler queue capacities set to 20,000 with updated comments. Batch sizes left unchanged: asnBatchSize (30,000) now exceeds its queue, but each handler flushes on its own timer regardless of batch fill, and one queued UPDATE contributes every ASN in its path, so flushing is unaffected and no size change is needed. No README text quoted the old figure. TODO.md not touched, matching the other memory-reduction units. Model: opus-4-8

clawbot closed this issue

2026-09-21 16:29:39 +02:00

clawbot referenced this issue from a commit

2026-09-21 16:29:40 +02:00

Shrink the four handler queues from 100,000 to 20,000 (closes #11)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: sneak/routewatch#11