GetRandomPrefixesByLengthContext in internal/database/database.go, behind /prefixlength/{length} and /prefixlength6/{length}, groups every live route with that mask length by prefix to find each prefix's newest time, shuffles all of those prefixes, keeps 500, then reads the routes of that length a second time to join them back. Its cost grows with the number of routes of that length, not with the 500 it shows.
Measured on a test database of one million /24 routes, with the file in memory: about 0.5 s a request. The 24-hour run on #3 ended with a 9.4 GiB database, about 25 million routes at the size per route of that test database (estimated, mostly IPv4 /24), still growing about 0.35 GiB an hour. /prefixlength/24 is heading for the 30-second request limit, sooner when the file is not in memory.
The join also returns every route of a chosen prefix whose time equals the newest. Times are whole seconds and a change reaches many peers in the same second, so a prefix can be listed several times, and the page's count includes the repeats. On the test database, with 40 peers sharing each prefix's time, the 500 prefixes came back as 20,000 rows.
Severity: medium
Definition of done
A test, failing before the fix: two peers' routes for one prefix with the same last_updated appear once in GetRandomPrefixesByLength.
The sample is drawn with a bounded number of indexed lookups, independent of the number of routes, and lists each prefix once.
A check on a database of about 2 million live routes, built within a 2-3 GiB memory budget: both pages answer well under a second.
make check passes.
Model: opus-5-5
`GetRandomPrefixesByLengthContext` in `internal/database/database.go`, behind `/prefixlength/{length}` and `/prefixlength6/{length}`, groups every live route with that mask length by prefix to find each prefix's newest time, shuffles all of those prefixes, keeps 500, then reads the routes of that length a second time to join them back. Its cost grows with the number of routes of that length, not with the 500 it shows.
Measured on a test database of one million /24 routes, with the file in memory: about 0.5 s a request. The 24-hour run on https://git.eeqj.de/sneak/routewatch/issues/3 ended with a 9.4 GiB database, about 25 million routes at the size per route of that test database (estimated, mostly IPv4 /24), still growing about 0.35 GiB an hour. `/prefixlength/24` is heading for the 30-second request limit, sooner when the file is not in memory.
The join also returns every route of a chosen prefix whose time equals the newest. Times are whole seconds and a change reaches many peers in the same second, so a prefix can be listed several times, and the page's count includes the repeats. On the test database, with 40 peers sharing each prefix's time, the 500 prefixes came back as 20,000 rows.
Severity: medium
## Definition of done
- A test, failing before the fix: two peers' routes for one prefix with the same `last_updated` appear once in `GetRandomPrefixesByLength`.
- The sample is drawn with a bounded number of indexed lookups, independent of the number of routes, and lists each prefix once.
- A check on a database of about 2 million live routes, built within a 2-3 GiB memory budget: both pages answer well under a second.
- `make check` passes.
Model: opus-5-5
clawbot
self-assigned this 2026-10-14 14:50:31 +02:00
Built in #79. A mask length with at most 500 prefixes now has all of them listed. A longer one is sampled with random row-id lookups on the mask length index, at most four per prefix asked for. Each prefix is listed once, with its newest route. The sample is uneven: prefixes with more routes are picked more often.
Model: opus-5-5
Built in https://git.eeqj.de/sneak/routewatch/pulls/79. A mask length with at most 500 prefixes now has all of them listed. A longer one is sampled with random row-id lookups on the mask length index, at most four per prefix asked for. Each prefix is listed once, with its newest route. The sample is uneven: prefixes with more routes are picked more often.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
GetRandomPrefixesByLengthContextininternal/database/database.go, behind/prefixlength/{length}and/prefixlength6/{length}, groups every live route with that mask length by prefix to find each prefix's newest time, shuffles all of those prefixes, keeps 500, then reads the routes of that length a second time to join them back. Its cost grows with the number of routes of that length, not with the 500 it shows.Measured on a test database of one million /24 routes, with the file in memory: about 0.5 s a request. The 24-hour run on #3 ended with a 9.4 GiB database, about 25 million routes at the size per route of that test database (estimated, mostly IPv4 /24), still growing about 0.35 GiB an hour.
/prefixlength/24is heading for the 30-second request limit, sooner when the file is not in memory.The join also returns every route of a chosen prefix whose time equals the newest. Times are whole seconds and a change reaches many peers in the same second, so a prefix can be listed several times, and the page's count includes the repeats. On the test database, with 40 peers sharing each prefix's time, the 500 prefixes came back as 20,000 rows.
Severity: medium
Definition of done
last_updatedappear once inGetRandomPrefixesByLength.make checkpasses.Model: opus-5-5
Built in #79. A mask length with at most 500 prefixes now has all of them listed. A longer one is sampled with random row-id lookups on the mask length index, at most four per prefix asked for. Each prefix is listed once, with its newest route. The sample is uneven: prefixes with more routes are picked more often.
Model: opus-5-5