Cap glibc malloc arenas to keep non-Go RSS bounded (closes #23)
check / check (push) Failing after 1s
check / check (push) Failing after 1s
After the earlier memory work, SQLite's live heap is bounded but process RSS still climbed about 70 MiB/min, nearly all anonymous and outside the Go runtime and outside SQLite's own accounting, without any SQLITE_NOMEM. The cause is glibc: the SQLite C library allocates and frees millions of small page-cache chunks from many threads, and glibc keeps each arena's freed chunks resident. With arenas uncapped it creates up to eight per core, so on a large host RSS grows with the core count. Set MALLOC_ARENA_MAX=2 in the image to bound the retained memory. Writes are already serialized, so the two-arena cap costs no throughput. Update the README Memory section. Model: opus-4-8
This commit is contained in:
@@ -83,6 +83,14 @@ ENV XDG_DATA_HOME=/var/lib
|
||||
# XDG_DATA_HOME above.
|
||||
ENV GOMEMLIMIT=1536MiB
|
||||
|
||||
# Cap glibc's malloc arenas. The SQLite C library allocates and frees millions
|
||||
# of small page-cache chunks from many threads; glibc otherwise creates up to
|
||||
# eight arenas per core (hundreds on a large host) and keeps each arena's freed
|
||||
# chunks resident, so process RSS climbs far above SQLite's live heap and never
|
||||
# comes back down. Two arenas keep that retained memory bounded; database writes
|
||||
# are already serialized, so the lost allocator concurrency costs nothing here.
|
||||
ENV MALLOC_ARENA_MAX=2
|
||||
|
||||
# Expose HTTP port
|
||||
EXPOSE 8080
|
||||
|
||||
|
||||
@@ -169,6 +169,7 @@ Configuration is handled via environment variables and OS-specific paths:
|
||||
| `PORT` | `8080` | HTTP server port |
|
||||
| `DEBUG` | (empty) | Set to `routewatch` for debug logging |
|
||||
| `GOMEMLIMIT` | `1536MiB` (in the Docker image) | Go soft memory limit; see Memory |
|
||||
| `MALLOC_ARENA_MAX` | `2` (in the Docker image) | glibc malloc arena cap; see Memory |
|
||||
|
||||
State directory (database location):
|
||||
- macOS: `~/Library/Application Support/routewatch/`
|
||||
@@ -183,6 +184,13 @@ Budget:
|
||||
- Go heap: a 1.5 GiB soft limit (`GOMEMLIMIT=1536MiB`, set in the image).
|
||||
- SQLite: at most 640 MiB of page cache across the connection pool (64 MiB per
|
||||
connection, 10 connections) and a 1.5 GiB hard heap limit for the C library.
|
||||
- glibc allocator: the SQLite C library runs on glibc `malloc`, which frees
|
||||
page-cache chunks back to per-arena free lists rather than to the kernel, so
|
||||
process RSS tracks the high-water mark of those arenas, not SQLite's live
|
||||
heap. glibc creates up to eight arenas per core, so on a many-core host the
|
||||
retained memory — and thus RSS — grows with the core count. The image sets
|
||||
`MALLOC_ARENA_MAX=2` to bound it; the two-arena cap costs nothing here because
|
||||
database writes are already serialized.
|
||||
- About 0.2 GiB for everything else in the runtime.
|
||||
|
||||
Run the container with a memory limit of 5 GiB and swap disabled:
|
||||
@@ -195,7 +203,9 @@ or the equivalent in your deployment tool. This leaves headroom above the
|
||||
ceilings for spikes and the kernel page cache.
|
||||
|
||||
Override the Go soft limit by setting `GOMEMLIMIT` in the environment (for
|
||||
example `-e GOMEMLIMIT=1GiB`); this replaces the image default.
|
||||
example `-e GOMEMLIMIT=1GiB`); this replaces the image default. `MALLOC_ARENA_MAX`
|
||||
can be overridden the same way, but raising it lets RSS climb again on a
|
||||
many-core host.
|
||||
|
||||
What happens at each limit:
|
||||
- Go soft limit: as the heap approaches `GOMEMLIMIT`, the runtime runs garbage
|
||||
|
||||
Reference in New Issue
Block a user