Write batches through one connection with a 1 GiB cache, in key order (closes #96)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
Every write now uses one connection taken from the pool when the database opens and used under the database lock. Its 1 GiB page cache keeps the pages the route writes touch from one batch to the next. A reading connection's cache is emptied whenever another connection has written, so the 9 reading connections drop to 16 MiB each. The soft and hard heap limits rise to 1.5 and 2 GiB so they do not shrink the caches, and README "Memory" adds up the new budget. Batch writes sort their rows by peer, then prefix. The benchmark also times the largest prefix handler batch, 250,000 updates. Model: opus-5-5
This commit is contained in:
@@ -117,7 +117,7 @@ routewatch/
|
||||
- Signal handlers registered for graceful shutdown
|
||||
|
||||
2. **Initialization** (`internal/routewatch/app.go`)
|
||||
- Database created with SQLite schema (WAL mode, 3GB cache)
|
||||
- Database created with SQLite schema (WAL mode, page caches as in "Memory")
|
||||
- Message handlers registered with the streamer
|
||||
- HTTP server started on configured port
|
||||
|
||||
@@ -215,8 +215,12 @@ data it tracks. The image sets ceilings that keep it inside a 5 GiB container.
|
||||
Budget:
|
||||
|
||||
- Go heap: a 1.5 GiB soft limit (`GOMEMLIMIT=1536MiB`, set in the image).
|
||||
- SQLite: at most 640 MiB of page cache across the connection pool (64 MiB per
|
||||
connection, 10 connections) and a 1.5 GiB hard heap limit for the C library.
|
||||
- SQLite: 1 GiB of page cache for the one connection every write uses, which
|
||||
keeps the pages the route writes touch from one batch to the next, and 16 MiB
|
||||
for each of the 9 reading connections, whose cache SQLite empties whenever
|
||||
another connection has written: about 1.15 GiB in all. A 1.5 GiB soft heap
|
||||
limit sits above the caches, so it does not shrink them, and a 2 GiB hard heap
|
||||
limit caps the C library.
|
||||
- glibc allocator: the SQLite C library runs on glibc `malloc`, which frees
|
||||
page-cache chunks back to per-arena free lists rather than to the kernel, so
|
||||
process RSS tracks the high-water mark of those arenas, not SQLite's live
|
||||
@@ -226,6 +230,9 @@ Budget:
|
||||
database writes are already serialized.
|
||||
- About 0.2 GiB for everything else in the runtime.
|
||||
|
||||
The ceilings add up to 3.7 GiB: the Go heap's 1.5 GiB, SQLite's 2 GiB and the
|
||||
0.2 GiB for the rest.
|
||||
|
||||
Run the container with a memory limit of 5 GiB and swap disabled:
|
||||
|
||||
```bash
|
||||
@@ -244,8 +251,8 @@ What happens at each limit:
|
||||
|
||||
- Go soft limit: as the heap approaches `GOMEMLIMIT`, the runtime runs garbage
|
||||
collection more aggressively rather than growing further.
|
||||
- SQLite: at a 1 GiB soft heap limit it recycles its page cache instead of
|
||||
allocating more; at the 1.5 GiB hard heap limit a statement fails with an
|
||||
- SQLite: at the 1.5 GiB soft heap limit it recycles its page cache instead of
|
||||
allocating more; at the 2 GiB hard heap limit a statement fails with an
|
||||
out-of-memory error, and the handler logs it and drops that batch. The process
|
||||
keeps running.
|
||||
- Handler queues: each of the four handler queues holds at most 20,000 messages.
|
||||
|
||||
Reference in New Issue
Block a user