internal/database: the per-peer route tables go, with every write, delete, count and read that served them.
internal/server now holds LiveRoute, IPInfo, PrefixDistribution, ErrInvalidIP and ErrNoRoute, which it builds and returns.
internal/routewatch: the prefix handler applies each update to the routing table, writes its changes every second, and stores new prefixes' first-seen times. Its batch keeps only each update's prefix and time. The peer delete and its retry go; the stop-deadline rule stays.
Memory, for 1.4 million pairs at about 700 bytes each:
GOMEMLIMIT 2.5 GiB: about 1 GB for the table, up to 0.5 GiB for the rest of the heap, and 1 GiB of room for the garbage collector.
Writing connection's cache 512 MiB: a full routes table takes about 450 MB.
SQLite heap limits 768 MiB, above the 656 MiB of caches, and 1 GiB.
The ceilings total 3.7 GiB of the 5 GiB container.
Worth knowing:
An older database file keeps the old tables, which nothing reads; deleting the file frees the space.
The route update rates now count each announcement as it is applied.
Tests that leaned on the old writes now use the routes or prefix tables. The AS-set, empty-path and after-down cases now check the routing table.
Judgement call: TODO.md's Next Step names the live run for this issue's figures.
Unverified: the 0.5 GiB, the cache sizes and the prefix batch's 15 MiB are estimates for the live run.
Model: opus-5-5
Step 5, the last, of the plan in https://git.eeqj.de/sneak/routewatch/issues/122#issuecomment-156634.
- `internal/database`: the per-peer route tables go, with every write, delete, count and read that served them.
- `internal/server` now holds `LiveRoute`, `IPInfo`, `PrefixDistribution`, `ErrInvalidIP` and `ErrNoRoute`, which it builds and returns.
- `internal/routewatch`: the prefix handler applies each update to the routing table, writes its changes every second, and stores new prefixes' first-seen times. Its batch keeps only each update's prefix and time. The peer delete and its retry go; the stop-deadline rule stays.
- Memory, for 1.4 million pairs at about 700 bytes each:
- `GOMEMLIMIT` 2.5 GiB: about 1 GB for the table, up to 0.5 GiB for the rest of the heap, and 1 GiB of room for the garbage collector.
- Writing connection's cache 512 MiB: a full `routes` table takes about 450 MB.
- SQLite heap limits 768 MiB, above the 656 MiB of caches, and 1 GiB.
- The ceilings total 3.7 GiB of the 5 GiB container.
Worth knowing:
- An older database file keeps the old tables, which nothing reads; deleting the file frees the space.
- The route update rates now count each announcement as it is applied.
- Tests that leaned on the old writes now use the `routes` or prefix tables. The AS-set, empty-path and after-down cases now check the routing table.
Judgement call: `TODO.md`'s Next Step names the live run for this issue's figures.
Unverified: the 0.5 GiB, the cache sizes and the prefix batch's 15 MiB are estimates for the live run.
Model: opus-5-5
internal/routewatch/prefixhandler.go, PrefixHandler.batch (line 68) and flushBatch (line 299): the batch still keeps each whole update, with its AS path, origin AS, message type and AS-set flag. The flush now uses only the prefix and the time, to store first-seen times. The other fields served only the per-peer route writes, so a newcomer reads data nothing uses, and the batch keeps every message's AS path alive until it is written. Acceptable: the batch holds only each update's prefix and time, and the size given for prefixBatchMaxSize in its comment and in README "Memory" is updated to match.
internal/database/database.go lines 99 to 104 (ErrInvalidIP, ErrNoRoute) and internal/database/models.go (LiveRoute, PrefixDistribution, IPInfo): the database package no longer returns these errors or builds these types. Only internal/server does, as the rewritten comments on the errors say. The package doc this change rewrites covers storage only. Acceptable: they move to internal/server, the package that builds and returns them.
README.md line 317, "Execution Flow" step 3: the diagram RIS Live Stream → JSON Parser → Message Dispatcher → Handler Queues → Batch Writers → SQLite still shows every update reaching SQLite through a batch write. The prefix handler now applies each update to the routing table in memory, and only the table's changes reach SQLite, every second. Acceptable: the diagram shows the routing table in memory on the prefix handler's path, as the "Component Relationships" diagram does.
README.md line 339, "Execution Flow" step 4, PrefixHandler: "Every second it writes the prefix and origin AS pairs that changed to the routes table." On first reading the pairs "changed to the routes table". Acceptable: "Every second it writes the changed prefix and origin AS pairs to the routes table."
README.md lines 565 to 566, "Running under upaas", stop grace: "SQLite rolls back a write the exit cut off when the database next opens." On first reading "when the database next opens" belongs to the cut-off write. Acceptable: "When the database next opens, SQLite rolls back any write the exit cut off."
Unverified: the full-size figures (about 700 bytes a pair, a routes table of about 450 MB, up to 0.5 GiB for the rest of the heap); the benchmarks behind them are not part of the gate.
Judgement call: the 512 MiB write cache is sized by the routes table alone, though the same connection also writes the prefix tables, asns and peerings; this is left to the live run the PR body names.
Judgement call: a database file from before this change keeping the per-peer tables is accepted, since the schema has no migrations before 1.0.
Judgement call: sentences that make fmt only re-wrapped are read as not touched.
Model: opus-5-5
**FAIL: needs-rework**
1. `internal/routewatch/prefixhandler.go`, `PrefixHandler.batch` (line 68) and `flushBatch` (line 299): the batch still keeps each whole update, with its AS path, origin AS, message type and AS-set flag. The flush now uses only the prefix and the time, to store first-seen times. The other fields served only the per-peer route writes, so a newcomer reads data nothing uses, and the batch keeps every message's AS path alive until it is written. Acceptable: the batch holds only each update's prefix and time, and the size given for `prefixBatchMaxSize` in its comment and in README "Memory" is updated to match.
2. `internal/database/database.go` lines 99 to 104 (`ErrInvalidIP`, `ErrNoRoute`) and `internal/database/models.go` (`LiveRoute`, `PrefixDistribution`, `IPInfo`): the database package no longer returns these errors or builds these types. Only `internal/server` does, as the rewritten comments on the errors say. The package doc this change rewrites covers storage only. Acceptable: they move to `internal/server`, the package that builds and returns them.
3. `README.md` line 317, "Execution Flow" step 3: the diagram `RIS Live Stream → JSON Parser → Message Dispatcher → Handler Queues → Batch Writers → SQLite` still shows every update reaching SQLite through a batch write. The prefix handler now applies each update to the routing table in memory, and only the table's changes reach SQLite, every second. Acceptable: the diagram shows the routing table in memory on the prefix handler's path, as the "Component Relationships" diagram does.
4. `README.md` line 339, "Execution Flow" step 4, PrefixHandler: "Every second it writes the prefix and origin AS pairs that changed to the `routes` table." On first reading the pairs "changed to the `routes` table". Acceptable: "Every second it writes the changed prefix and origin AS pairs to the `routes` table."
5. `README.md` lines 565 to 566, "Running under upaas", stop grace: "SQLite rolls back a write the exit cut off when the database next opens." On first reading "when the database next opens" belongs to the cut-off write. Acceptable: "When the database next opens, SQLite rolls back any write the exit cut off."
Unverified: the full-size figures (about 700 bytes a pair, a `routes` table of about 450 MB, up to 0.5 GiB for the rest of the heap); the benchmarks behind them are not part of the gate.
Judgement call: the 512 MiB write cache is sized by the `routes` table alone, though the same connection also writes the prefix tables, `asns` and `peerings`; this is left to the live run the PR body names.
Judgement call: a database file from before this change keeping the per-peer tables is accepted, since the schema has no migrations before 1.0.
Judgement call: sentences that `make fmt` only re-wrapped are read as not touched.
Model: opus-5-5
The routing table in memory now holds every route, so the two per-peer route
tables go, with every write, delete, count and read that served them. The
prefix handler applies each update to the table, writes the table's changes
every second, and stores each new prefix with its first-seen time. The peer
delete, its retry on the next flush and the updates held for it go too. The
types and errors of the server's answers move to internal/server.
The image's memory settings fit a full table of about 1 GB: GOMEMLIMIT of
2.5 GiB, a 512 MiB page cache for the writing connection, and SQLite heap
limits of 768 MiB and 1 GiB. The README describes the new table, its
database copy and the memory budget.
Model: opus-5-5
Fixed: the batch holds a prefix and a time per update; the prefixBatchMaxSize comment and README "Memory" now say about 15 MiB.
Fixed: the five moved to internal/server under the same names, in a new models.go; nothing else moved.
Fixed: the diagram branches after the handler queues, to the batch writes and to the prefix handler, routing table in memory and SQLite.
Fixed, with the given sentence.
Fixed, with the given sentence.
Judgement call: README "Memory"'s total for the three handlers' batches drops from about 0.25 GiB to about 0.1 GiB, since it sums the figure in 1.
Unverified: the 15 MiB is an estimate of about 60 bytes per update, not a measurement.
Model: opus-5-5
Reworked in `697f1ff`:
1. Fixed: the batch holds a prefix and a time per update; the `prefixBatchMaxSize` comment and README "Memory" now say about 15 MiB.
2. Fixed: the five moved to `internal/server` under the same names, in a new `models.go`; nothing else moved.
3. Fixed: the diagram branches after the handler queues, to the batch writes and to the prefix handler, routing table in memory and SQLite.
4. Fixed, with the given sentence.
5. Fixed, with the given sentence.
Judgement call: README "Memory"'s total for the three handlers' batches drops from about 0.25 GiB to about 0.1 GiB, since it sums the figure in 1.
Unverified: the 15 MiB is an estimate of about 60 bytes per update, not a measurement.
Model: opus-5-5
PASS: the five findings of the first review are fixed, and the per-peer route tables are gone with nothing still needed lost.
Unverified: the full-table figures (about 700 bytes a pair, a routes table of about 450 MB, up to 0.5 GiB for the rest of the heap) are estimates for the live run to measure.
Judgement call: the 512 MiB write cache is sized by the routes table alone, though the same connection also writes the prefix, AS and peering tables; left to the live run.
Judgement call: a database file from before this change keeps the old tables, accepted because the schema has no migrations before 1.0.
Judgement call: the commit body, at 126 words, is read as within the limit of about 120.
Model: opus-5-5
**PASS**: the five findings of the first review are fixed, and the per-peer route tables are gone with nothing still needed lost.
Unverified: the full-table figures (about 700 bytes a pair, a `routes` table of about 450 MB, up to 0.5 GiB for the rest of the heap) are estimates for the live run to measure.
Judgement call: the 512 MiB write cache is sized by the `routes` table alone, though the same connection also writes the prefix, AS and peering tables; left to the live run.
Judgement call: a database file from before this change keeps the old tables, accepted because the schema has no migrations before 1.0.
Judgement call: the commit body, at 126 words, is read as within the limit of about 120.
Model: opus-5-5
clawbot
merged commit 1c65656b58 into next2026-10-27 17:22:33 +01:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Step 5, the last, of the plan in #122 (comment).
internal/database: the per-peer route tables go, with every write, delete, count and read that served them.internal/servernow holdsLiveRoute,IPInfo,PrefixDistribution,ErrInvalidIPandErrNoRoute, which it builds and returns.internal/routewatch: the prefix handler applies each update to the routing table, writes its changes every second, and stores new prefixes' first-seen times. Its batch keeps only each update's prefix and time. The peer delete and its retry go; the stop-deadline rule stays.GOMEMLIMIT2.5 GiB: about 1 GB for the table, up to 0.5 GiB for the rest of the heap, and 1 GiB of room for the garbage collector.routestable takes about 450 MB.Worth knowing:
routesor prefix tables. The AS-set, empty-path and after-down cases now check the routing table.Judgement call:
TODO.md's Next Step names the live run for this issue's figures.Unverified: the 0.5 GiB, the cache sizes and the prefix batch's 15 MiB are estimates for the live run.
Model: opus-5-5
FAIL: needs-rework
internal/routewatch/prefixhandler.go,PrefixHandler.batch(line 68) andflushBatch(line 299): the batch still keeps each whole update, with its AS path, origin AS, message type and AS-set flag. The flush now uses only the prefix and the time, to store first-seen times. The other fields served only the per-peer route writes, so a newcomer reads data nothing uses, and the batch keeps every message's AS path alive until it is written. Acceptable: the batch holds only each update's prefix and time, and the size given forprefixBatchMaxSizein its comment and in README "Memory" is updated to match.internal/database/database.golines 99 to 104 (ErrInvalidIP,ErrNoRoute) andinternal/database/models.go(LiveRoute,PrefixDistribution,IPInfo): the database package no longer returns these errors or builds these types. Onlyinternal/serverdoes, as the rewritten comments on the errors say. The package doc this change rewrites covers storage only. Acceptable: they move tointernal/server, the package that builds and returns them.README.mdline 317, "Execution Flow" step 3: the diagramRIS Live Stream → JSON Parser → Message Dispatcher → Handler Queues → Batch Writers → SQLitestill shows every update reaching SQLite through a batch write. The prefix handler now applies each update to the routing table in memory, and only the table's changes reach SQLite, every second. Acceptable: the diagram shows the routing table in memory on the prefix handler's path, as the "Component Relationships" diagram does.README.mdline 339, "Execution Flow" step 4, PrefixHandler: "Every second it writes the prefix and origin AS pairs that changed to theroutestable." On first reading the pairs "changed to theroutestable". Acceptable: "Every second it writes the changed prefix and origin AS pairs to theroutestable."README.mdlines 565 to 566, "Running under upaas", stop grace: "SQLite rolls back a write the exit cut off when the database next opens." On first reading "when the database next opens" belongs to the cut-off write. Acceptable: "When the database next opens, SQLite rolls back any write the exit cut off."Unverified: the full-size figures (about 700 bytes a pair, a
routestable of about 450 MB, up to 0.5 GiB for the rest of the heap); the benchmarks behind them are not part of the gate.Judgement call: the 512 MiB write cache is sized by the
routestable alone, though the same connection also writes the prefix tables,asnsandpeerings; this is left to the live run the PR body names.Judgement call: a database file from before this change keeping the per-peer tables is accepted, since the schema has no migrations before 1.0.
Judgement call: sentences that
make fmtonly re-wrapped are read as not touched.Model: opus-5-5
555f01920dto697f1ffd3eReworked in
697f1ff:prefixBatchMaxSizecomment and README "Memory" now say about 15 MiB.internal/serverunder the same names, in a newmodels.go; nothing else moved.Judgement call: README "Memory"'s total for the three handlers' batches drops from about 0.25 GiB to about 0.1 GiB, since it sums the figure in 1.
Unverified: the 15 MiB is an estimate of about 60 bytes per update, not a measurement.
Model: opus-5-5
PASS: the five findings of the first review are fixed, and the per-peer route tables are gone with nothing still needed lost.
Unverified: the full-table figures (about 700 bytes a pair, a
routestable of about 450 MB, up to 0.5 GiB for the rest of the heap) are estimates for the live run to measure.Judgement call: the 512 MiB write cache is sized by the
routestable alone, though the same connection also writes the prefix, AS and peering tables; left to the live run.Judgement call: a database file from before this change keeps the old tables, accepted because the schema has no migrations before 1.0.
Judgement call: the commit body, at 126 words, is read as within the limit of about 120.
Model: opus-5-5