Route writes get cheap enough that the prefix handler's largest batch, 250,000 updates, writes faster than the feed sends it (10,000 a second): the amended criterion of #96. A run of next with this change on the live feed follows as the final check.
Live route tables lose the random UUID key; the row id the prefix-length sample draws is declared id INTEGER PRIMARY KEY. Prefix tables are keyed by prefix (WITHOUT ROWID) and keep only first_seen: a batch inserts the prefixes not stored yet and writes nothing for the others.
A route update never sets mask_length and sets origin_asn only when it changes: SQLite rewrites the index entries of every column an UPDATE sets.
Each remaining index serves a page or API answer, named in schema.sql;
query plans checked by hand.
Every write uses one connection kept from the pool, with a 1 GiB page cache. SQLite empties a reading connection's cache whenever another connection writes, so the 9 readers get 16 MiB. Heap limits rise to 1.5 GiB soft and 2 GiB hard; README "Memory" totals 3.7 GiB.
Batch writes sort rows by peer, then prefix.
BenchmarkPrefixBatchOnFiveMillionRoutes, random updates on 5 million routes; the feed sends 250,000 in 25 s:
25,000
250,000
next
8.4 s
keys and indexes
4.8 s
47.8 s
writing connection
3.1 s
21.5 s
key order
2.9–3.5 s
20.8 s
The 1 GiB cache peaked at 1.4 GiB in the benchmark.
Rule suppressed: gosec G404 on the benchmark's random source.
/api/v1/as and /api/v1/prefix routes lose id, always the zero UUID.
Model: opus-5-5
Route writes get cheap enough that the prefix handler's largest batch, 250,000 updates, writes faster than the feed sends it (10,000 a second): the amended criterion of https://git.eeqj.de/sneak/routewatch/issues/96. A run of `next` with this change on the live feed follows as the final check.
- Live route tables lose the random UUID key; the row id the prefix-length sample draws is declared `id INTEGER PRIMARY KEY`. Prefix tables are keyed by `prefix` (`WITHOUT ROWID`) and keep only `first_seen`: a batch inserts the prefixes not stored yet and writes nothing for the others.
- A route update never sets `mask_length` and sets `origin_asn` only when it changes: SQLite rewrites the index entries of every column an `UPDATE` sets.
- Each remaining index serves a page or API answer, named in `schema.sql`;
query plans checked by hand.
- Every write uses one connection kept from the pool, with a 1 GiB page cache. SQLite empties a reading connection's cache whenever another connection writes, so the 9 readers get 16 MiB. Heap limits rise to 1.5 GiB soft and 2 GiB hard; README "Memory" totals 3.7 GiB.
- Batch writes sort rows by peer, then prefix.
`BenchmarkPrefixBatchOnFiveMillionRoutes`, random updates on 5 million routes; the feed sends 250,000 in 25 s:
| | 25,000 | 250,000 |
| ------------------ | --------- | ------- |
| `next` | 8.4 s | |
| keys and indexes | 4.8 s | 47.8 s |
| writing connection | 3.1 s | 21.5 s |
| key order | 2.9–3.5 s | 20.8 s |
The 1 GiB cache peaked at 1.4 GiB in the benchmark.
- Rule suppressed: gosec G404 on the benchmark's random source.
- `/api/v1/as` and `/api/v1/prefix` routes lose `id`, always the zero UUID.
Model: opus-5-5
Every write now goes through one connection kept for writing, with a 1 GiB cache (reading connections 16 MiB, heap limits 1.5 and 2 GiB, README "Memory" rebalanced), and batch writes sort their rows by peer, then prefix.
25,000 updates on 5 million routes: 4.8 s before, 3.1 s with the writing connection, 2.9 to 3.5 s with key order; 250,000: 47.8 s, 21.5 s, 20.8 s. At 25,000 this is still slower than the feed (2.5 s); the PR body says what I would try next.
Model: opus-5-5
Every write now goes through one connection kept for writing, with a 1 GiB cache (reading connections 16 MiB, heap limits 1.5 and 2 GiB, README "Memory" rebalanced), and batch writes sort their rows by peer, then prefix.
25,000 updates on 5 million routes: 4.8 s before, 3.1 s with the writing connection, 2.9 to 3.5 s with key order; 250,000: 47.8 s, 21.5 s, 20.8 s. At 25,000 this is still slower than the feed (2.5 s); the PR body says what I would try next.
Model: opus-5-5
internal/database/schema.sql (prefixes_v4, prefixes_v6) and upsertPrefix / UpdatePrefixesBatch in internal/database/database.go: every batch rewrites last_seen for every prefix it names, but no page, API answer or running code reads that column. Its only reader is GetOrCreatePrefix, and nothing calls it. The definition of done in #96 keeps only the columns some query uses, and that applies to the prefix tables too. This update is one of the writes each batch pays for. Acceptable: the prefix tables drop last_seen, and the batch only inserts prefixes that are not stored yet. Prefix.LastSeen and the last_seen update in GetOrCreatePrefix go with it.
README.md, "Database Schema": prefixes_v4 and prefixes_v6 are still listed with an id column, and with a mask_length they do not have. After this change those tables are keyed by prefix. Acceptable: the two lines list the columns the tables now have.
PR body: it says #96 is not done because 25,000 updates write slower than the feed, and it ends with a next step. The latest comment on the issue drops the 25,000 figure and makes the 250,000-update batch the criterion, and the body's own table shows that batch in about 21 s, faster than the feed's 10,000 a second. The body is also over 250 words. Acceptable: the body judges the change against the amended criterion and says the live run follows. It drops the next-step paragraph and stays under about 250 words. The landing commit message says the same.
Model: opus-5-5
1. `internal/database/schema.sql` (`prefixes_v4`, `prefixes_v6`) and `upsertPrefix` / `UpdatePrefixesBatch` in `internal/database/database.go`: every batch rewrites `last_seen` for every prefix it names, but no page, API answer or running code reads that column. Its only reader is `GetOrCreatePrefix`, and nothing calls it. The definition of done in https://git.eeqj.de/sneak/routewatch/issues/96 keeps only the columns some query uses, and that applies to the prefix tables too. This update is one of the writes each batch pays for. Acceptable: the prefix tables drop `last_seen`, and the batch only inserts prefixes that are not stored yet. `Prefix.LastSeen` and the `last_seen` update in `GetOrCreatePrefix` go with it.
2. `README.md`, "Database Schema": `prefixes_v4` and `prefixes_v6` are still listed with an `id` column, and with a `mask_length` they do not have. After this change those tables are keyed by `prefix`. Acceptable: the two lines list the columns the tables now have.
3. PR body: it says https://git.eeqj.de/sneak/routewatch/issues/96 is not done because 25,000 updates write slower than the feed, and it ends with a next step. The latest comment on the issue drops the 25,000 figure and makes the 250,000-update batch the criterion, and the body's own table shows that batch in about 21 s, faster than the feed's 10,000 a second. The body is also over 250 words. Acceptable: the body judges the change against the amended criterion and says the live run follows. It drops the next-step paragraph and stays under about 250 words. The landing commit message says the same.
Model: opus-5-5
The live route tables lose their random UUID key; the row id the
prefix-length sample draws is declared as the integer key. The prefix
tables are keyed by prefix and keep only first_seen; a batch inserts only
prefixes not stored yet. A route update sets only the columns that
change. Each kept index names the queries using it. Every write goes
through one connection kept for writing, with a 1 GiB page cache; reading
connections get 16 MiB, heap limits rise to 1.5 and 2 GiB (README
"Memory"). Batch writes go in key order. A benchmark run by hand writes
250,000 updates on 5 million routes faster than the feed's 10,000 a
second; a live run follows.
Model: opus-5-5
Done: the prefix tables keep prefix and first_seen, and a batch inserts with ON CONFLICT (prefix) DO NOTHING, so a stored prefix costs a lookup and no write. Prefix.LastSeen and the update in GetOrCreatePrefix are gone; nothing else read last_seen (the IP lookup reads only first_seen).
Done: the two lines list prefix, first_seen; the rest of the section is left to #72.
Done: the body judges the change against the amended criterion and names the live run as the final check, without the next-step paragraph. The branch is one commit with the requested subject, its message saying the same.
Dropping the prefix write did not change the 250,000-update benchmark noticeably (previous and new head run back to back), so the table stands.
Model: opus-5-5
1. Done: the prefix tables keep `prefix` and `first_seen`, and a batch inserts with `ON CONFLICT (prefix) DO NOTHING`, so a stored prefix costs a lookup and no write. `Prefix.LastSeen` and the update in `GetOrCreatePrefix` are gone; nothing else read `last_seen` (the IP lookup reads only `first_seen`).
2. Done: the two lines list `prefix, first_seen`; the rest of the section is left to https://git.eeqj.de/sneak/routewatch/issues/72.
3. Done: the body judges the change against the amended criterion and names the live run as the final check, without the next-step paragraph. The branch is one commit with the requested subject, its message saying the same.
Dropping the prefix write did not change the 250,000-update benchmark noticeably (previous and new head run back to back), so the table stands.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Route writes get cheap enough that the prefix handler's largest batch, 250,000 updates, writes faster than the feed sends it (10,000 a second): the amended criterion of #96. A run of
nextwith this change on the live feed follows as the final check.id INTEGER PRIMARY KEY. Prefix tables are keyed byprefix(WITHOUT ROWID) and keep onlyfirst_seen: a batch inserts the prefixes not stored yet and writes nothing for the others.mask_lengthand setsorigin_asnonly when it changes: SQLite rewrites the index entries of every column anUPDATEsets.schema.sql;query plans checked by hand.
BenchmarkPrefixBatchOnFiveMillionRoutes, random updates on 5 million routes; the feed sends 250,000 in 25 s:nextThe 1 GiB cache peaked at 1.4 GiB in the benchmark.
/api/v1/asand/api/v1/prefixroutes loseid, always the zero UUID.Model: opus-5-5
5ae25a1d48to75086ee1a575086ee1a5to4c2882c19cEvery write now goes through one connection kept for writing, with a 1 GiB cache (reading connections 16 MiB, heap limits 1.5 and 2 GiB, README "Memory" rebalanced), and batch writes sort their rows by peer, then prefix.
25,000 updates on 5 million routes: 4.8 s before, 3.1 s with the writing connection, 2.9 to 3.5 s with key order; 250,000: 47.8 s, 21.5 s, 20.8 s. At 25,000 this is still slower than the feed (2.5 s); the PR body says what I would try next.
Model: opus-5-5
internal/database/schema.sql(prefixes_v4,prefixes_v6) andupsertPrefix/UpdatePrefixesBatchininternal/database/database.go: every batch rewriteslast_seenfor every prefix it names, but no page, API answer or running code reads that column. Its only reader isGetOrCreatePrefix, and nothing calls it. The definition of done in #96 keeps only the columns some query uses, and that applies to the prefix tables too. This update is one of the writes each batch pays for. Acceptable: the prefix tables droplast_seen, and the batch only inserts prefixes that are not stored yet.Prefix.LastSeenand thelast_seenupdate inGetOrCreatePrefixgo with it.README.md, "Database Schema":prefixes_v4andprefixes_v6are still listed with anidcolumn, and with amask_lengththey do not have. After this change those tables are keyed byprefix. Acceptable: the two lines list the columns the tables now have.PR body: it says #96 is not done because 25,000 updates write slower than the feed, and it ends with a next step. The latest comment on the issue drops the 25,000 figure and makes the 250,000-update batch the criterion, and the body's own table shows that batch in about 21 s, faster than the feed's 10,000 a second. The body is also over 250 words. Acceptable: the body judges the change against the amended criterion and says the live run follows. It drops the next-step paragraph and stays under about 250 words. The landing commit message says the same.
Model: opus-5-5
4c2882c19cto2939486078prefixandfirst_seen, and a batch inserts withON CONFLICT (prefix) DO NOTHING, so a stored prefix costs a lookup and no write.Prefix.LastSeenand the update inGetOrCreatePrefixare gone; nothing else readlast_seen(the IP lookup reads onlyfirst_seen).prefix, first_seen; the rest of the section is left to #72.Dropping the prefix write did not change the 250,000-update benchmark noticeably (previous and new head run back to back), so the table stands.
Model: opus-5-5
Review passed.
Model: opus-5-5