Cap glibc malloc arenas to keep non-Go RSS bounded (closes #23) #24

Merged
clawbot merged 1 commits from issue-23-malloc-arena into next 2026-09-21 19:29:32 +02:00
Collaborator

Sets MALLOC_ARENA_MAX=2 in the runtime image so process memory stops climbing
on the production feed.

After the earlier memory work (#3),
SQLite's live heap is bounded but RSS still grew about 70 MiB/min, nearly all
anonymous, outside both the Go runtime and SQLite's own accounting, with no
SQLITE_NOMEM. Cause: glibc. SQLite frees millions of small page-cache chunks
from many threads; with arenas uncapped glibc makes up to eight per core
(hundreds on this 48-core host) and keeps each arena's freed chunks resident.
Capping to two arenas bounds the retained memory; writes are already serialized,
so the cap costs no throughput.

Only the Dockerfile and the README Memory section change; no code.

Verification (disclosure): two containers from the same 658aadb build on the
live feed for 36 minutes, identical but for MALLOC_ARENA_MAX. Non-Go RSS
(RssAnon minus the Go runtime's Sys minus HeapReleased): uncapped rose
531 to 1152 MiB and kept climbing with the database; the fixed image levelled at
196-285 MiB for the whole run, under 1.2 GiB with wide margin. Full figures on
#23.

Disclosures:

  • Ran without --memory (host refuses it); RSS from /proc with a 6 GiB
    watchdog, so behaviour at the container limit is untested.
  • The baseline arm reused the existing 658aadb image; the two arms differ only
    by the arena env.
  • database is locked still occurs; cause noted on the issue, not fixed here per
    its instruction.

Model: opus-4-8

Sets `MALLOC_ARENA_MAX=2` in the runtime image so process memory stops climbing on the production feed. After the earlier memory work (https://git.eeqj.de/sneak/routewatch/issues/3), SQLite's live heap is bounded but RSS still grew about 70 MiB/min, nearly all anonymous, outside both the Go runtime and SQLite's own accounting, with no `SQLITE_NOMEM`. Cause: glibc. SQLite frees millions of small page-cache chunks from many threads; with arenas uncapped glibc makes up to eight per core (hundreds on this 48-core host) and keeps each arena's freed chunks resident. Capping to two arenas bounds the retained memory; writes are already serialized, so the cap costs no throughput. Only the `Dockerfile` and the README Memory section change; no code. Verification (disclosure): two containers from the same `658aadb` build on the live feed for 36 minutes, identical but for `MALLOC_ARENA_MAX`. Non-Go RSS (RssAnon minus the Go runtime's `Sys` minus `HeapReleased`): uncapped rose 531 to 1152 MiB and kept climbing with the database; the fixed image levelled at 196-285 MiB for the whole run, under 1.2 GiB with wide margin. Full figures on https://git.eeqj.de/sneak/routewatch/issues/23. Disclosures: - Ran without `--memory` (host refuses it); RSS from `/proc` with a 6 GiB watchdog, so behaviour at the container limit is untested. - The baseline arm reused the existing `658aadb` image; the two arms differ only by the arena env. - `database is locked` still occurs; cause noted on the issue, not fixed here per its instruction. Model: opus-4-8
clawbot added 1 commit 2026-09-21 19:06:27 +02:00
After the earlier memory work, SQLite's live heap is bounded but process
RSS still climbed about 70 MiB/min, nearly all anonymous and outside the
Go runtime and outside SQLite's own accounting, without any SQLITE_NOMEM.
The cause is glibc: the SQLite C library allocates and frees millions of
small page-cache chunks from many threads, and glibc keeps each arena's
freed chunks resident. With arenas uncapped it creates up to eight per
core, so on a large host RSS grows with the core count.

Set MALLOC_ARENA_MAX=2 in the image to bound the retained memory. Writes
are already serialized, so the two-arena cap costs no throughput. Update
the README Memory section.

Model: opus-4-8
clawbot added the needs-review label 2026-09-21 19:06:36 +02:00
clawbot self-assigned this 2026-09-21 19:06:36 +02:00
Author
Collaborator

PASS: MALLOC_ARENA_MAX=2 is set in the glibc runtime stage and reaches the routewatch process through the entrypoint's runuser, the diagnosis on #23 supports the fix, every changed README sentence holds, and the Docker build gate is green.

Model: opus-4-8

PASS: `MALLOC_ARENA_MAX=2` is set in the glibc runtime stage and reaches the routewatch process through the entrypoint's `runuser`, the diagnosis on https://git.eeqj.de/sneak/routewatch/issues/23 supports the fix, every changed README sentence holds, and the Docker build gate is green. Model: opus-4-8
clawbot removed the needs-review label 2026-09-21 19:21:44 +02:00
clawbot merged commit 08f060045b into next 2026-09-21 19:29:32 +02:00
clawbot deleted branch issue-23-malloc-arena 2026-09-21 19:29:32 +02:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/routewatch#24