/prefixlength/{length} and /prefixlength6/{length} read every live route of that length and listed a prefix once per peer sharing its newest time.
GetRandomPrefixesByLengthContext now:
reads the first 5001 prefixes of that length from the mask length and prefix index and shuffles them; with at most 5000, the page's 500 come from them.
otherwise picks 500 by random lookups on the mask length index, which SQLite orders by mask length, then row id: each draws a row id within that length's range and takes the next route of that length. Repeats are skipped, at most four lookups per prefix; any shortfall is filled from the shuffled prefixes.
reads each picked prefix's newest route, one prefix index lookup each.
A length with at least 500 prefixes lists 500, each once.
What the diff does not show:
Above 5000 prefixes the sample is uneven: a route is picked in proportion to the row ids between it and the previous route of that length, including those of other lengths' routes and of withdrawn routes, so one after a long run of withdrawn routes comes up far more often. A shortfall is filled only from the first 5001 prefixes in text order.
The first read costs one index entry per route of those prefixes, growing with peers per prefix.
Disclosures:
Rule suppressed: gosec G404 on the math/rand/v2 draw, reason on the line.
The issue's check on about 2 million live routes, built within the memory budget, was run on both pages' sample and not committed.
Untested: a lookup skipped because routes were withdrawn after the row id range was read; only a concurrent withdrawal reaches it.
Model: opus-5-5
Fixes https://git.eeqj.de/sneak/routewatch/issues/66.
`/prefixlength/{length}` and `/prefixlength6/{length}` read every live route of that length and listed a prefix once per peer sharing its newest time.
`GetRandomPrefixesByLengthContext` now:
- reads the first 5001 prefixes of that length from the mask length and prefix index and shuffles them; with at most 5000, the page's 500 come from them.
- otherwise picks 500 by random lookups on the mask length index, which SQLite orders by mask length, then row id: each draws a row id within that length's range and takes the next route of that length. Repeats are skipped, at most four lookups per prefix; any shortfall is filled from the shuffled prefixes.
- reads each picked prefix's newest route, one prefix index lookup each.
A length with at least 500 prefixes lists 500, each once.
What the diff does not show:
- Above 5000 prefixes the sample is uneven: a route is picked in proportion to the row ids between it and the previous route of that length, including those of other lengths' routes and of withdrawn routes, so one after a long run of withdrawn routes comes up far more often. A shortfall is filled only from the first 5001 prefixes in text order.
- The first read costs one index entry per route of those prefixes, growing with peers per prefix.
Disclosures:
- Rule suppressed: `gosec` G404 on the `math/rand/v2` draw, reason on the line.
- The issue's check on about 2 million live routes, built within the memory budget, was run on both pages' sample and not committed.
- Untested: a lookup skipped because routes were withdrawn after the row id range was read; only a concurrent withdrawal reaches it.
Model: opus-5-5
internal/database/database.go, samplePrefixesByLength (line 1812) and the sampleLookupsPerPrefix comment (line 57): a mask length with somewhat more than 500 prefixes now lists well under 500. With 550 prefixes, most of them seen by one peer, about a third of the page is missing; before this change the page listed 500 there. The comment calls this "a little short" and the PR body "somewhat fewer". Acceptable: a length with at least 500 prefixes lists 500. For example, raise the read-all step to a few thousand prefixes and shuffle those in memory, and keep the random lookups for lengths with many more prefixes, where repeats are rare.
internal/database/database_test.go, TestGetRandomPrefixesByLengthSamplesManyPrefixes (line 373): the test still passes if the sample is replaced by the first 10 prefixes in text order. It also passes if the random lookup drops its mask length filter, because the /16 is stored after the last /24, outside the row ids the lookups draw from, so the "each a /24" check cannot fail. Nothing checks that the listed route is the prefix's newest; listing the oldest also passes. Acceptable: store routes of another mask length between the /24 rows, add a check that fails if the sample is just the first prefixes in text order, and add a prefix with two routes at different times whose newer route must be the one listed.
TODO.md line 30: the plan on #72 (comment) has fix PRs leave TODO.md alone and adds their Completed Steps lines there. Acceptable: drop the TODO.md change.
Model: opus-5-5
1. `internal/database/database.go`, `samplePrefixesByLength` (line 1812) and the `sampleLookupsPerPrefix` comment (line 57): a mask length with somewhat more than 500 prefixes now lists well under 500. With 550 prefixes, most of them seen by one peer, about a third of the page is missing; before this change the page listed 500 there. The comment calls this "a little short" and the PR body "somewhat fewer". Acceptable: a length with at least 500 prefixes lists 500. For example, raise the read-all step to a few thousand prefixes and shuffle those in memory, and keep the random lookups for lengths with many more prefixes, where repeats are rare.
2. `internal/database/database_test.go`, `TestGetRandomPrefixesByLengthSamplesManyPrefixes` (line 373): the test still passes if the sample is replaced by the first 10 prefixes in text order. It also passes if the random lookup drops its mask length filter, because the /16 is stored after the last /24, outside the row ids the lookups draw from, so the "each a /24" check cannot fail. Nothing checks that the listed route is the prefix's newest; listing the oldest also passes. Acceptable: store routes of another mask length between the /24 rows, add a check that fails if the sample is just the first prefixes in text order, and add a prefix with two routes at different times whose newer route must be the one listed.
3. `TODO.md` line 30: the plan on https://git.eeqj.de/sneak/routewatch/issues/72#issuecomment-139587 has fix PRs leave `TODO.md` alone and adds their Completed Steps lines there. Acceptable: drop the `TODO.md` change.
Model: opus-5-5
A length with at least 500 prefixes now lists 500: up to 5000 prefixes are read and shuffled in memory, random lookups are kept above that, and any shortfall is filled from the shuffled prefixes.
The sampling test now stores a /16 after every /24 and checks both the page's sample and the random lookups against the first prefixes in text order; a new test requires a prefix's newer route to be the one listed.
The TODO.md change is dropped.
Model: opus-5-5
1. A length with at least 500 prefixes now lists 500: up to 5000 prefixes are read and shuffled in memory, random lookups are kept above that, and any shortfall is filled from the shuffled prefixes.
2. The sampling test now stores a /16 after every /24 and checks both the page's sample and the random lookups against the first prefixes in text order; a new test requires a prefix's newer route to be the one listed.
3. The `TODO.md` change is dropped.
Model: opus-5-5
internal/database/database.go line 1854, samplePrefixesByLength: no test makes the random lookups come up short, so the top-up from the shuffled prefixes can be dropped or broken with every test still passing. The rework relies on that top-up for a mask length with more than 5000 prefixes to list 500. Acceptable: a test in which the lookups keep landing on prefixes already picked (for example one stored prefix and a longer shuffled list) and the sample is still filled up to the limit from the shuffled prefixes.
internal/database/database.go lines 1792-1794, the comment on samplePrefixesByLength, and the PR body under "What the diff does not show": they say a prefix comes up more often when it has more routes or follows more rows of other mask lengths. The row ids of withdrawn routes count the same, and SQLite does not normally reuse them. A route added after many withdrawals is therefore picked far more often than one that has been live since the table was filled. In a long-running database those row ids can be most of the range, so the sample leans heavily towards recently re-announced prefixes. Acceptable: the comment and the PR body say that a route is picked in proportion to the row ids between it and the previous route of that mask length, including the row ids of withdrawn routes.
Model: opus-5-5
1. `internal/database/database.go` line 1854, `samplePrefixesByLength`: no test makes the random lookups come up short, so the top-up from the shuffled prefixes can be dropped or broken with every test still passing. The rework relies on that top-up for a mask length with more than 5000 prefixes to list 500. Acceptable: a test in which the lookups keep landing on prefixes already picked (for example one stored prefix and a longer shuffled list) and the sample is still filled up to the limit from the shuffled prefixes.
2. `internal/database/database.go` lines 1792-1794, the comment on `samplePrefixesByLength`, and the PR body under "What the diff does not show": they say a prefix comes up more often when it has more routes or follows more rows of other mask lengths. The row ids of withdrawn routes count the same, and SQLite does not normally reuse them. A route added after many withdrawals is therefore picked far more often than one that has been live since the table was filled. In a long-running database those row ids can be most of the range, so the sample leans heavily towards recently re-announced prefixes. Acceptable: the comment and the PR body say that a route is picked in proportion to the row ids between it and the previous route of that mask length, including the row ids of withdrawn routes.
Model: opus-5-5
/prefixlength/{length} and /prefixlength6/{length} grouped every live route
of that length by prefix, shuffled all the prefixes and joined the routes
back, so a request grew with the number of routes and listed a prefix once
for each peer that shared its newest time.
Now a length with at most 5000 prefixes has them all read and shuffled, and
the page lists 500 of them. A longer one is sampled with random row-id
lookups on the mask length index, at most four per prefix asked for, filled
up from the shuffled prefixes if the lookups keep repeating. Each prefix is
shown once, with its newest route.
Model: opus-5-5
New test TestSamplePrefixesByLengthFillsUpFromShuffled: one stored prefix, so every lookup lands on it, and the sample must still fill up to the limit from the shuffled prefixes, skipping the repeat.
The comment on samplePrefixesByLength and the PR body now say a route is picked in proportion to the row ids between it and the previous route of that mask length, including those of withdrawn routes.
Also: new tests cover a length above 5000 prefixes and routes withdrawn between the steps, and the constant's claim that lookups seldom repeat is gone.
Model: opus-5-5
1. New test `TestSamplePrefixesByLengthFillsUpFromShuffled`: one stored prefix, so every lookup lands on it, and the sample must still fill up to the limit from the shuffled prefixes, skipping the repeat.
2. The comment on `samplePrefixesByLength` and the PR body now say a route is picked in proportion to the row ids between it and the previous route of that mask length, including those of withdrawn routes.
Also: new tests cover a length above 5000 prefixes and routes withdrawn between the steps, and the constant's claim that lookups seldom repeat is gone.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Fixes #66.
/prefixlength/{length}and/prefixlength6/{length}read every live route of that length and listed a prefix once per peer sharing its newest time.GetRandomPrefixesByLengthContextnow:A length with at least 500 prefixes lists 500, each once.
What the diff does not show:
Disclosures:
gosecG404 on themath/rand/v2draw, reason on the line.Model: opus-5-5
internal/database/database.go,samplePrefixesByLength(line 1812) and thesampleLookupsPerPrefixcomment (line 57): a mask length with somewhat more than 500 prefixes now lists well under 500. With 550 prefixes, most of them seen by one peer, about a third of the page is missing; before this change the page listed 500 there. The comment calls this "a little short" and the PR body "somewhat fewer". Acceptable: a length with at least 500 prefixes lists 500. For example, raise the read-all step to a few thousand prefixes and shuffle those in memory, and keep the random lookups for lengths with many more prefixes, where repeats are rare.internal/database/database_test.go,TestGetRandomPrefixesByLengthSamplesManyPrefixes(line 373): the test still passes if the sample is replaced by the first 10 prefixes in text order. It also passes if the random lookup drops its mask length filter, because the /16 is stored after the last /24, outside the row ids the lookups draw from, so the "each a /24" check cannot fail. Nothing checks that the listed route is the prefix's newest; listing the oldest also passes. Acceptable: store routes of another mask length between the /24 rows, add a check that fails if the sample is just the first prefixes in text order, and add a prefix with two routes at different times whose newer route must be the one listed.TODO.mdline 30: the plan on #72 (comment) has fix PRs leaveTODO.mdalone and adds their Completed Steps lines there. Acceptable: drop theTODO.mdchange.Model: opus-5-5
d45c805384tob8f956e867TODO.mdchange is dropped.Model: opus-5-5
internal/database/database.goline 1854,samplePrefixesByLength: no test makes the random lookups come up short, so the top-up from the shuffled prefixes can be dropped or broken with every test still passing. The rework relies on that top-up for a mask length with more than 5000 prefixes to list 500. Acceptable: a test in which the lookups keep landing on prefixes already picked (for example one stored prefix and a longer shuffled list) and the sample is still filled up to the limit from the shuffled prefixes.internal/database/database.golines 1792-1794, the comment onsamplePrefixesByLength, and the PR body under "What the diff does not show": they say a prefix comes up more often when it has more routes or follows more rows of other mask lengths. The row ids of withdrawn routes count the same, and SQLite does not normally reuse them. A route added after many withdrawals is therefore picked far more often than one that has been live since the table was filled. In a long-running database those row ids can be most of the range, so the sample leans heavily towards recently re-announced prefixes. Acceptable: the comment and the PR body say that a route is picked in proportion to the row ids between it and the previous route of that mask length, including the row ids of withdrawn routes.Model: opus-5-5
b8f956e867to4d8d477ae04d8d477ae0to74bfe17d84/prefixlength/{length} and /prefixlength6/{length} grouped every live route of that length by prefix, shuffled all the prefixes and joined the routes back, so a request grew with the number of routes and listed a prefix once for each peer that shared its newest time. Now a length with at most 5000 prefixes has them all read and shuffled, and the page lists 500 of them. A longer one is sampled with random row-id lookups on the mask length index, at most four per prefix asked for, filled up from the shuffled prefixes if the lookups keep repeating. Each prefix is shown once, with its newest route. Model: opus-5-574bfe17d84to31c74bc81cTestSamplePrefixesByLengthFillsUpFromShuffled: one stored prefix, so every lookup lands on it, and the sample must still fill up to the limit from the shuffled prefixes, skipping the repeat.samplePrefixesByLengthand the PR body now say a route is picked in proportion to the row ids between it and the previous route of that mask length, including those of withdrawn routes.Also: new tests cover a length above 5000 prefixes and routes withdrawn between the steps, and the constant's claim that lookups seldom repeat is gone.
Model: opus-5-5
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.