Write batches through one connection with a 1 GiB cache, in key order (closes #96)
check / check (push) Canceled after 0s

Every write now uses one connection taken from the pool when the
database opens, under the database lock. Its 1 GiB page cache keeps
the pages the route writes touch from one batch to the next. A reading
connection's cache is emptied whenever another connection has written,
so the 9 reading connections drop to 16 MiB each. The soft and hard
heap limits rise to 1.5 and 2 GiB, above the caches; README "Memory"
adds up the new budget. Vacuum, Analyze and Checkpoint check their
context after the lock, which the kept connection does not. Batch
writes sort their rows by peer, then prefix. The benchmark also times
the largest prefix handler batch, 250,000 updates.

Model: opus-5-5
This commit is contained in:
2026-10-19 12:52:03 +00:00
parent 1f612642e3
commit 75086ee1a5
4 changed files with 269 additions and 73 deletions
+12 -5
View File
@@ -117,7 +117,7 @@ routewatch/
- Signal handlers registered for graceful shutdown
2. **Initialization** (`internal/routewatch/app.go`)
- Database created with SQLite schema (WAL mode, 3GB cache)
- Database created with SQLite schema (WAL mode, page caches as in "Memory")
- Message handlers registered with the streamer
- HTTP server started on configured port
@@ -215,8 +215,12 @@ data it tracks. The image sets ceilings that keep it inside a 5 GiB container.
Budget:
- Go heap: a 1.5 GiB soft limit (`GOMEMLIMIT=1536MiB`, set in the image).
- SQLite: at most 640 MiB of page cache across the connection pool (64 MiB per
connection, 10 connections) and a 1.5 GiB hard heap limit for the C library.
- SQLite: 1 GiB of page cache for the one connection every write uses, which
keeps the pages the route writes touch from one batch to the next, and 16 MiB
for each of the 9 reading connections, whose cache SQLite empties whenever
another connection has written: about 1.15 GiB in all. A 1.5 GiB soft heap
limit sits above the caches, so it does not shrink them, and a 2 GiB hard heap
limit caps the C library.
- glibc allocator: the SQLite C library runs on glibc `malloc`, which frees
page-cache chunks back to per-arena free lists rather than to the kernel, so
process RSS tracks the high-water mark of those arenas, not SQLite's live
@@ -226,6 +230,9 @@ Budget:
database writes are already serialized.
- About 0.2 GiB for everything else in the runtime.
The ceilings add up to 3.7 GiB: the Go heap's 1.5 GiB, SQLite's 2 GiB and the
0.2 GiB for the rest.
Run the container with a memory limit of 5 GiB and swap disabled:
```bash
@@ -244,8 +251,8 @@ What happens at each limit:
- Go soft limit: as the heap approaches `GOMEMLIMIT`, the runtime runs garbage
collection more aggressively rather than growing further.
- SQLite: at a 1 GiB soft heap limit it recycles its page cache instead of
allocating more; at the 1.5 GiB hard heap limit a statement fails with an
- SQLite: at the 1.5 GiB soft heap limit it recycles its page cache instead of
allocating more; at the 2 GiB hard heap limit a statement fails with an
out-of-memory error, and the handler logs it and drops that batch. The process
keeps running.
- Handler queues: each of the four handler queues holds at most 20,000 messages.