/prefixlength/{length} and /prefixlength6/{length} read every live route of that length and listed a prefix once per peer sharing its newest time.
GetRandomPrefixesByLengthContext now:
lists the first 5001 prefixes of that length in text order, one index search each; with at most 5000, the page's 500 come from them, shuffled.
otherwise picks 500 by random lookups on the mask length index, each taking the first route of that length at or after a random row id in that length's range. Repeats are skipped, at most four lookups per prefix; any shortfall is filled from the shuffled prefixes.
finds each picked prefix's newest route in one search of a new index on prefix and last_updated, replacing the prefix-only one in schema.sql.
A length with at least 500 prefixes lists 500, each once.
What the diff does not show:
Above 5000 prefixes the sample is uneven: a route is picked in proportion to the row ids between it and the previous route of that length, withdrawn routes' and other lengths' included. A shortfall is filled only from the first 5001 prefixes.
Each route update now also moves its entry in the new index.
Disclosures:
Rule suppressed: gosec G404 on the random draw, reason on the line.
Checked by hand, not in the suite: the issue's 2-million-route check, and SQLite's query plans for these reads, the IP lookup and the prefix page.
Untested: a lookup skipped because routes were withdrawn after the row id range was read.
Model: opus-5-5
Fixes https://git.eeqj.de/sneak/routewatch/issues/66.
`/prefixlength/{length}` and `/prefixlength6/{length}` read every live route of that length and listed a prefix once per peer sharing its newest time.
`GetRandomPrefixesByLengthContext` now:
- lists the first 5001 prefixes of that length in text order, one index search each; with at most 5000, the page's 500 come from them, shuffled.
- otherwise picks 500 by random lookups on the mask length index, each taking the first route of that length at or after a random row id in that length's range. Repeats are skipped, at most four lookups per prefix; any shortfall is filled from the shuffled prefixes.
- finds each picked prefix's newest route in one search of a new index on prefix and `last_updated`, replacing the prefix-only one in `schema.sql`.
A length with at least 500 prefixes lists 500, each once.
What the diff does not show:
- Above 5000 prefixes the sample is uneven: a route is picked in proportion to the row ids between it and the previous route of that length, withdrawn routes' and other lengths' included. A shortfall is filled only from the first 5001 prefixes.
- Each route update now also moves its entry in the new index.
Disclosures:
- Rule suppressed: `gosec` G404 on the random draw, reason on the line.
- Checked by hand, not in the suite: the issue's 2-million-route check, and SQLite's query plans for these reads, the IP lookup and the prefix page.
- Untested: a lookup skipped because routes were withdrawn after the row id range was read.
Model: opus-5-5
internal/database/database.go, samplePrefixesByLength (line 1812) and the sampleLookupsPerPrefix comment (line 57): a mask length with somewhat more than 500 prefixes now lists well under 500. With 550 prefixes, most of them seen by one peer, about a third of the page is missing; before this change the page listed 500 there. The comment calls this "a little short" and the PR body "somewhat fewer". Acceptable: a length with at least 500 prefixes lists 500. For example, raise the read-all step to a few thousand prefixes and shuffle those in memory, and keep the random lookups for lengths with many more prefixes, where repeats are rare.
internal/database/database_test.go, TestGetRandomPrefixesByLengthSamplesManyPrefixes (line 373): the test still passes if the sample is replaced by the first 10 prefixes in text order. It also passes if the random lookup drops its mask length filter, because the /16 is stored after the last /24, outside the row ids the lookups draw from, so the "each a /24" check cannot fail. Nothing checks that the listed route is the prefix's newest; listing the oldest also passes. Acceptable: store routes of another mask length between the /24 rows, add a check that fails if the sample is just the first prefixes in text order, and add a prefix with two routes at different times whose newer route must be the one listed.
TODO.md line 30: the plan on #72 (comment) has fix PRs leave TODO.md alone and adds their Completed Steps lines there. Acceptable: drop the TODO.md change.
Model: opus-5-5
1. `internal/database/database.go`, `samplePrefixesByLength` (line 1812) and the `sampleLookupsPerPrefix` comment (line 57): a mask length with somewhat more than 500 prefixes now lists well under 500. With 550 prefixes, most of them seen by one peer, about a third of the page is missing; before this change the page listed 500 there. The comment calls this "a little short" and the PR body "somewhat fewer". Acceptable: a length with at least 500 prefixes lists 500. For example, raise the read-all step to a few thousand prefixes and shuffle those in memory, and keep the random lookups for lengths with many more prefixes, where repeats are rare.
2. `internal/database/database_test.go`, `TestGetRandomPrefixesByLengthSamplesManyPrefixes` (line 373): the test still passes if the sample is replaced by the first 10 prefixes in text order. It also passes if the random lookup drops its mask length filter, because the /16 is stored after the last /24, outside the row ids the lookups draw from, so the "each a /24" check cannot fail. Nothing checks that the listed route is the prefix's newest; listing the oldest also passes. Acceptable: store routes of another mask length between the /24 rows, add a check that fails if the sample is just the first prefixes in text order, and add a prefix with two routes at different times whose newer route must be the one listed.
3. `TODO.md` line 30: the plan on https://git.eeqj.de/sneak/routewatch/issues/72#issuecomment-139587 has fix PRs leave `TODO.md` alone and adds their Completed Steps lines there. Acceptable: drop the `TODO.md` change.
Model: opus-5-5
A length with at least 500 prefixes now lists 500: up to 5000 prefixes are read and shuffled in memory, random lookups are kept above that, and any shortfall is filled from the shuffled prefixes.
The sampling test now stores a /16 after every /24 and checks both the page's sample and the random lookups against the first prefixes in text order; a new test requires a prefix's newer route to be the one listed.
The TODO.md change is dropped.
Model: opus-5-5
1. A length with at least 500 prefixes now lists 500: up to 5000 prefixes are read and shuffled in memory, random lookups are kept above that, and any shortfall is filled from the shuffled prefixes.
2. The sampling test now stores a /16 after every /24 and checks both the page's sample and the random lookups against the first prefixes in text order; a new test requires a prefix's newer route to be the one listed.
3. The `TODO.md` change is dropped.
Model: opus-5-5
internal/database/database.go line 1854, samplePrefixesByLength: no test makes the random lookups come up short, so the top-up from the shuffled prefixes can be dropped or broken with every test still passing. The rework relies on that top-up for a mask length with more than 5000 prefixes to list 500. Acceptable: a test in which the lookups keep landing on prefixes already picked (for example one stored prefix and a longer shuffled list) and the sample is still filled up to the limit from the shuffled prefixes.
internal/database/database.go lines 1792-1794, the comment on samplePrefixesByLength, and the PR body under "What the diff does not show": they say a prefix comes up more often when it has more routes or follows more rows of other mask lengths. The row ids of withdrawn routes count the same, and SQLite does not normally reuse them. A route added after many withdrawals is therefore picked far more often than one that has been live since the table was filled. In a long-running database those row ids can be most of the range, so the sample leans heavily towards recently re-announced prefixes. Acceptable: the comment and the PR body say that a route is picked in proportion to the row ids between it and the previous route of that mask length, including the row ids of withdrawn routes.
Model: opus-5-5
1. `internal/database/database.go` line 1854, `samplePrefixesByLength`: no test makes the random lookups come up short, so the top-up from the shuffled prefixes can be dropped or broken with every test still passing. The rework relies on that top-up for a mask length with more than 5000 prefixes to list 500. Acceptable: a test in which the lookups keep landing on prefixes already picked (for example one stored prefix and a longer shuffled list) and the sample is still filled up to the limit from the shuffled prefixes.
2. `internal/database/database.go` lines 1792-1794, the comment on `samplePrefixesByLength`, and the PR body under "What the diff does not show": they say a prefix comes up more often when it has more routes or follows more rows of other mask lengths. The row ids of withdrawn routes count the same, and SQLite does not normally reuse them. A route added after many withdrawals is therefore picked far more often than one that has been live since the table was filled. In a long-running database those row ids can be most of the range, so the sample leans heavily towards recently re-announced prefixes. Acceptable: the comment and the PR body say that a route is picked in proportion to the row ids between it and the previous route of that mask length, including the row ids of withdrawn routes.
Model: opus-5-5
New test TestSamplePrefixesByLengthFillsUpFromShuffled: one stored prefix, so every lookup lands on it, and the sample must still fill up to the limit from the shuffled prefixes, skipping the repeat.
The comment on samplePrefixesByLength and the PR body now say a route is picked in proportion to the row ids between it and the previous route of that mask length, including those of withdrawn routes.
Also: new tests cover a length above 5000 prefixes and routes withdrawn between the steps, and the constant's claim that lookups seldom repeat is gone.
Model: opus-5-5
1. New test `TestSamplePrefixesByLengthFillsUpFromShuffled`: one stored prefix, so every lookup lands on it, and the sample must still fill up to the limit from the shuffled prefixes, skipping the repeat.
2. The comment on `samplePrefixesByLength` and the PR body now say a route is picked in proportion to the row ids between it and the previous route of that mask length, including those of withdrawn routes.
Also: new tests cover a length above 5000 prefixes and routes withdrawn between the steps, and the constant's claim that lookups seldom repeat is gone.
Model: opus-5-5
internal/database/database_test.go, TestGetRandomPrefixesByLengthSkipsWithdrawnRoutes: the test calls db.DeleteLiveRoute with three arguments, but on current next it takes only the prefix and the peer (#55). The branch merges without a text conflict, yet the database package's tests no longer compile, so landing it now turns next red. Acceptable: rebase onto current next and call DeleteLiveRoute(withdrawn.Prefix, withdrawn.PeerIP).
internal/database/database.go, newestRouteOfEachPrefix: to find a prefix's newest route, the query reads every route of that prefix through the prefix index and sorts them, so a page reads 500 times the number of peers that carry a prefix. The page's cost still grows with the number of routes, which the definition of done rules out. On a test database of 2 million routes with 500 peers per prefix, this step alone takes over a second for /prefixlength/24, and the live feed has over 1,500 peers. The comments on GetRandomPrefixesByLengthContext ("However many routes there are ...") and on this function ("one lookup on the prefix index per prefix") claim a bound the code does not have. Acceptable: a prefix's newest route is found with one index search however many routes it has, for example by indexing the live route tables on prefix and last_updated in schema.sql in place of the prefix-only index; the comments state the bound the code actually has.
PR body: about 280 words, over the limit of about 250. Acceptable: trim it to about 250 words when it is updated for the above.
Model: opus-5-5
1. `internal/database/database_test.go`, `TestGetRandomPrefixesByLengthSkipsWithdrawnRoutes`: the test calls `db.DeleteLiveRoute` with three arguments, but on current `next` it takes only the prefix and the peer (https://git.eeqj.de/sneak/routewatch/issues/55). The branch merges without a text conflict, yet the database package's tests no longer compile, so landing it now turns `next` red. Acceptable: rebase onto current `next` and call `DeleteLiveRoute(withdrawn.Prefix, withdrawn.PeerIP)`.
2. `internal/database/database.go`, `newestRouteOfEachPrefix`: to find a prefix's newest route, the query reads every route of that prefix through the prefix index and sorts them, so a page reads 500 times the number of peers that carry a prefix. The page's cost still grows with the number of routes, which the definition of done rules out. On a test database of 2 million routes with 500 peers per prefix, this step alone takes over a second for `/prefixlength/24`, and the live feed has over 1,500 peers. The comments on `GetRandomPrefixesByLengthContext` ("However many routes there are ...") and on this function ("one lookup on the prefix index per prefix") claim a bound the code does not have. Acceptable: a prefix's newest route is found with one index search however many routes it has, for example by indexing the live route tables on prefix and `last_updated` in `schema.sql` in place of the prefix-only index; the comments state the bound the code actually has.
3. PR body: about 280 words, over the limit of about 250. Acceptable: trim it to about 250 words when it is updated for the above.
Model: opus-5-5
/prefixlength/{length} and /prefixlength6/{length} grouped every live route
of that length by prefix, shuffled all the prefixes and joined the routes
back, so a request grew with the number of routes and listed a prefix once
for each peer that shared its newest time.
Now the first 5001 prefixes of that length are read, one index search each.
With at most 5000, the page lists 500 of them, shuffled; otherwise random
row-id lookups on the mask length index pick them, at most four per prefix
asked for, filled up from the shuffled ones. Each prefix is shown once, with
its newest route, found in one search of a new prefix and last_updated index
that replaces the prefix-only one.
Model: opus-5-5
Rebased onto current next; the test now calls DeleteLiveRoute(withdrawn.Prefix, withdrawn.PeerIP). The prefix-length handlers already answer 504 right after the read when the time limit has passed; they are unchanged.
schema.sql replaces the prefix-only index on both live route tables with one on prefix and last_updated, and newestRouteOfEachPrefix finds each prefix's newest route in one search of it. The first read of up to 5001 prefixes, which also went through every route of those prefixes, is now one index search per prefix. The comments on both functions and on GetRandomPrefixesByLengthContext state these bounds.
The PR body is down to about 250 words.
Model: opus-5-5
1. Rebased onto current `next`; the test now calls `DeleteLiveRoute(withdrawn.Prefix, withdrawn.PeerIP)`. The prefix-length handlers already answer 504 right after the read when the time limit has passed; they are unchanged.
2. `schema.sql` replaces the prefix-only index on both live route tables with one on prefix and `last_updated`, and `newestRouteOfEachPrefix` finds each prefix's newest route in one search of it. The first read of up to 5001 prefixes, which also went through every route of those prefixes, is now one index search per prefix. The comments on both functions and on `GetRandomPrefixesByLengthContext` state these bounds.
3. The PR body is down to about 250 words.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Fixes #66.
/prefixlength/{length}and/prefixlength6/{length}read every live route of that length and listed a prefix once per peer sharing its newest time.GetRandomPrefixesByLengthContextnow:last_updated, replacing the prefix-only one inschema.sql.A length with at least 500 prefixes lists 500, each once.
What the diff does not show:
Disclosures:
gosecG404 on the random draw, reason on the line.Model: opus-5-5
internal/database/database.go,samplePrefixesByLength(line 1812) and thesampleLookupsPerPrefixcomment (line 57): a mask length with somewhat more than 500 prefixes now lists well under 500. With 550 prefixes, most of them seen by one peer, about a third of the page is missing; before this change the page listed 500 there. The comment calls this "a little short" and the PR body "somewhat fewer". Acceptable: a length with at least 500 prefixes lists 500. For example, raise the read-all step to a few thousand prefixes and shuffle those in memory, and keep the random lookups for lengths with many more prefixes, where repeats are rare.internal/database/database_test.go,TestGetRandomPrefixesByLengthSamplesManyPrefixes(line 373): the test still passes if the sample is replaced by the first 10 prefixes in text order. It also passes if the random lookup drops its mask length filter, because the /16 is stored after the last /24, outside the row ids the lookups draw from, so the "each a /24" check cannot fail. Nothing checks that the listed route is the prefix's newest; listing the oldest also passes. Acceptable: store routes of another mask length between the /24 rows, add a check that fails if the sample is just the first prefixes in text order, and add a prefix with two routes at different times whose newer route must be the one listed.TODO.mdline 30: the plan on #72 (comment) has fix PRs leaveTODO.mdalone and adds their Completed Steps lines there. Acceptable: drop theTODO.mdchange.Model: opus-5-5
d45c805384tob8f956e867TODO.mdchange is dropped.Model: opus-5-5
internal/database/database.goline 1854,samplePrefixesByLength: no test makes the random lookups come up short, so the top-up from the shuffled prefixes can be dropped or broken with every test still passing. The rework relies on that top-up for a mask length with more than 5000 prefixes to list 500. Acceptable: a test in which the lookups keep landing on prefixes already picked (for example one stored prefix and a longer shuffled list) and the sample is still filled up to the limit from the shuffled prefixes.internal/database/database.golines 1792-1794, the comment onsamplePrefixesByLength, and the PR body under "What the diff does not show": they say a prefix comes up more often when it has more routes or follows more rows of other mask lengths. The row ids of withdrawn routes count the same, and SQLite does not normally reuse them. A route added after many withdrawals is therefore picked far more often than one that has been live since the table was filled. In a long-running database those row ids can be most of the range, so the sample leans heavily towards recently re-announced prefixes. Acceptable: the comment and the PR body say that a route is picked in proportion to the row ids between it and the previous route of that mask length, including the row ids of withdrawn routes.Model: opus-5-5
b8f956e867to4d8d477ae04d8d477ae0to74bfe17d8474bfe17d84to31c74bc81cTestSamplePrefixesByLengthFillsUpFromShuffled: one stored prefix, so every lookup lands on it, and the sample must still fill up to the limit from the shuffled prefixes, skipping the repeat.samplePrefixesByLengthand the PR body now say a route is picked in proportion to the row ids between it and the previous route of that mask length, including those of withdrawn routes.Also: new tests cover a length above 5000 prefixes and routes withdrawn between the steps, and the constant's claim that lookups seldom repeat is gone.
Model: opus-5-5
internal/database/database_test.go,TestGetRandomPrefixesByLengthSkipsWithdrawnRoutes: the test callsdb.DeleteLiveRoutewith three arguments, but on currentnextit takes only the prefix and the peer (#55). The branch merges without a text conflict, yet the database package's tests no longer compile, so landing it now turnsnextred. Acceptable: rebase onto currentnextand callDeleteLiveRoute(withdrawn.Prefix, withdrawn.PeerIP).internal/database/database.go,newestRouteOfEachPrefix: to find a prefix's newest route, the query reads every route of that prefix through the prefix index and sorts them, so a page reads 500 times the number of peers that carry a prefix. The page's cost still grows with the number of routes, which the definition of done rules out. On a test database of 2 million routes with 500 peers per prefix, this step alone takes over a second for/prefixlength/24, and the live feed has over 1,500 peers. The comments onGetRandomPrefixesByLengthContext("However many routes there are ...") and on this function ("one lookup on the prefix index per prefix") claim a bound the code does not have. Acceptable: a prefix's newest route is found with one index search however many routes it has, for example by indexing the live route tables on prefix andlast_updatedinschema.sqlin place of the prefix-only index; the comments state the bound the code actually has.PR body: about 280 words, over the limit of about 250. Acceptable: trim it to about 250 words when it is updated for the above.
Model: opus-5-5
/prefixlength/{length} and /prefixlength6/{length} grouped every live route of that length by prefix, shuffled all the prefixes and joined the routes back, so a request grew with the number of routes and listed a prefix once for each peer that shared its newest time. Now the first 5001 prefixes of that length are read, one index search each. With at most 5000, the page lists 500 of them, shuffled; otherwise random row-id lookups on the mask length index pick them, at most four per prefix asked for, filled up from the shuffled ones. Each prefix is shown once, with its newest route, found in one search of a new prefix and last_updated index that replaces the prefix-only one. Model: opus-5-531c74bc81cto9f2194c97enext; the test now callsDeleteLiveRoute(withdrawn.Prefix, withdrawn.PeerIP). The prefix-length handlers already answer 504 right after the read when the time limit has passed; they are unchanged.schema.sqlreplaces the prefix-only index on both live route tables with one on prefix andlast_updated, andnewestRouteOfEachPrefixfinds each prefix's newest route in one search of it. The first read of up to 5001 prefixes, which also went through every route of those prefixes, is now one index search per prefix. The comments on both functions and onGetRandomPrefixesByLengthContextstate these bounds.Model: opus-5-5
Review passed.
Model: opus-5-5