Run next on the live feed for 24 hours to confirm the fix for dropped updates #232

Closed
opened 2026-11-10 10:15:07 +01:00 by clawbot · 5 comments
Collaborator

Run next on the live feed for at least 24 hours, with an empty database, to confirm #229. The run of 15bf20a was stopped after 3 hours and the feed did not stall or reconnect during it, so the case that issue fixed never came up.

Commit. e58c663, the head of next, which also carries #161. Its origin_changes table has not been measured on the live feed.

Read from the run.

  • Whether any handler drops an update, and the prefix queue's high-water mark against its 100,000.
  • The held-back count against its limit of 150,000.
  • Memory, read from the process (this host cannot cap a container's memory) and compared with the 5 GiB in the README.
  • The size of origin_changes on disk, which the milestone PR says is not measured yet.

Done when. The run has lasted 24 hours, its outcome is a comment here, and each problem it found is its own issue, fixed through a reviewed PR. A feed stall or reconnect during the run is what confirms the fix; if none came up in 24 hours, the comment says so and the run goes on until one has, up to 72 hours in all.

Model: opus-5-5

Run `next` on the live feed for at least 24 hours, with an empty database, to confirm https://git.eeqj.de/sneak/routewatch/issues/229. The run of `15bf20a` was stopped after 3 hours and the feed did not stall or reconnect during it, so the case that issue fixed never came up. **Commit.** `e58c663`, the head of `next`, which also carries https://git.eeqj.de/sneak/routewatch/issues/161. Its `origin_changes` table has not been measured on the live feed. **Read from the run.** - Whether any handler drops an update, and the prefix queue's high-water mark against its 100,000. - The held-back count against its limit of 150,000. - Memory, read from the process (this host cannot cap a container's memory) and compared with the 5 GiB in the README. - The size of `origin_changes` on disk, which the milestone PR says is not measured yet. **Done when.** The run has lasted 24 hours, its outcome is a comment here, and each problem it found is its own issue, fixed through a reviewed PR. A feed stall or reconnect during the run is what confirms the fix; if none came up in 24 hours, the comment says so and the run goes on until one has, up to 72 hours in all. Model: opus-5-5
clawbot self-assigned this 2026-11-10 10:15:07 +01:00
Author
Collaborator

Running since 09:19 UTC on 10 November: container routewatch-run-e58c663, volume of the same name, 127.0.0.1:18431, statistics in the container log every 10 seconds and in /status.json. Whoever ends the run removes both.

Model: opus-5-5

Running since 09:19 UTC on 10 November: container `routewatch-run-e58c663`, volume of the same name, `127.0.0.1:18431`, statistics in the container log every 10 seconds and in `/status.json`. Whoever ends the run removes both. Model: opus-5-5
Author
Collaborator

The 24-hour live-feed run of next (container routewatch-run-e58c663) was stopped by the top-level manager after 31.5 hours: routewatch is outside the current focus list and no session was watching it. Its data volume routewatch-run-e58c663 is kept for the report.

Model: opus-5-5

The 24-hour live-feed run of `next` (container `routewatch-run-e58c663`) was stopped by the top-level manager after 31.5 hours: routewatch is outside the current focus list and no session was watching it. Its data volume `routewatch-run-e58c663` is kept for the report. Model: opus-5-5
clawbot added the needs-decision label 2026-11-11 20:40:23 +01:00
Author
Collaborator

No PR: this issue is a live-feed trial with no code change; the stopped run's container routewatch-run-e58c663 (exited, log intact) and its volume remain for the report.

Model: opus-5-5

No PR: this issue is a live-feed trial with no code change; the stopped run's container `routewatch-run-e58c663` (exited, log intact) and its volume remain for the report. Model: opus-5-5
Author
Collaborator

For the worker: no new run is needed. The run of e58c663 lasted 31.5 hours, more than the 24 this issue asks for, and its container routewatch-run-e58c663 (exited) and volume of the same name still hold its log and data.

  • Read the container log (docker logs routewatch-run-e58c663) and the database in the volume.
  • Post one report comment answering each point under "Read from the run" in the issue body, including whether the feed stalled or reconnected and whether any update was dropped.
  • If no update was dropped, close this issue and #229 if still open. If one was, file a bug with the evidence and leave this open.
  • Then remove the container and the volume.

Model: opus-5-5

For the worker: no new run is needed. The run of `e58c663` lasted 31.5 hours, more than the 24 this issue asks for, and its container `routewatch-run-e58c663` (exited) and volume of the same name still hold its log and data. - Read the container log (`docker logs routewatch-run-e58c663`) and the database in the volume. - Post one report comment answering each point under "Read from the run" in the issue body, including whether the feed stalled or reconnected and whether any update was dropped. - If no update was dropped, close this issue and https://git.eeqj.de/sneak/routewatch/issues/229 if still open. If one was, file a bug with the evidence and leave this open. - Then remove the container and the volume. Model: opus-5-5
clawbot removed the needs-decision label 2026-11-12 01:45:42 +01:00
Author
Collaborator

Report on the run of e58c663 (container routewatch-run-e58c663), from 09:19 UTC on 10 November to 16:48 UTC on 11 November: 31.5 hours, empty database at start. No handler dropped an update, and the feed disconnected and reconnected 68 times, so the fix for #229 is confirmed.

  • Dropped updates: none from any handler during the whole run.
  • Feed stalls and reconnects: 68 disconnects, 67 of them with scanner error: unexpected EOF and one with a truncated JSON line. Each reconnect succeeded within 5 to 20 seconds, with at most 3 failures in a row.
  • Prefix queue high-water mark: 60,961 of 100,000, at 14:20:13 UTC on 11 November, during ten seconds in which eight peers sent BGP notifications and their sessions reopened. The next highest was 44,216. The longest single message took 2.15 seconds. The other queues stayed under 3,200 of 20,000.
  • Held-back count: not recorded. The log does not carry it, and nobody saved the /status.json readings. At shutdown the handler held nothing back.
  • Memory: not recorded. The log has no memory figures, and the process is gone. Reading it needs a live run.
  • Size of origin_changes: 2.81 million rows. The table takes 101 MB and its prefix index 81 MB, out of an 888 MB database. Rows are kept for 72 hours, so the prune deleted none during the run. At the same rate, 72 hours would hold about 6.4 million rows, about 415 MB.

The run found no new problem. Container and volume removed.

Model: opus-5-5

Report on the run of `e58c663` (container `routewatch-run-e58c663`), from 09:19 UTC on 10 November to 16:48 UTC on 11 November: 31.5 hours, empty database at start. No handler dropped an update, and the feed disconnected and reconnected 68 times, so the fix for https://git.eeqj.de/sneak/routewatch/issues/229 is confirmed. - **Dropped updates:** none from any handler during the whole run. - **Feed stalls and reconnects:** 68 disconnects, 67 of them with `scanner error: unexpected EOF` and one with a truncated JSON line. Each reconnect succeeded within 5 to 20 seconds, with at most 3 failures in a row. - **Prefix queue high-water mark:** 60,961 of 100,000, at 14:20:13 UTC on 11 November, during ten seconds in which eight peers sent BGP notifications and their sessions reopened. The next highest was 44,216. The longest single message took 2.15 seconds. The other queues stayed under 3,200 of 20,000. - **Held-back count:** not recorded. The log does not carry it, and nobody saved the `/status.json` readings. At shutdown the handler held nothing back. - **Memory:** not recorded. The log has no memory figures, and the process is gone. Reading it needs a live run. - **Size of `origin_changes`:** 2.81 million rows. The table takes 101 MB and its prefix index 81 MB, out of an 888 MB database. Rows are kept for 72 hours, so the prune deleted none during the run. At the same rate, 72 hours would hold about 6.4 million rows, about 415 MB. The run found no new problem. Container and volume removed. Model: opus-5-5
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/routewatch#232