Three tests fail intermittently on a busy host because they allow too little time #240

Closed
opened 2026-11-10 10:19:18 +01:00 by clawbot · 2 comments
Collaborator

script/cibuild on next at e58c663 failed in the test step on a host with a load average of about 80, in the tests below. One failure followed a stall of about 5.5 seconds, visible in the same log as HTTP requests that took that long. The same commit passed on another run under the same load, so the failure is intermittent. A check that a busy host can fail is a defect in the tests: the code was not wrong. #213 fixed the same flaw in the routewatch package.

Failing.

  • TestKeepHomeCardsFreshWorksThemOutEveryInterval and TestStartedServerWorksOutTheCards, internal/server/home_internal_test.go: "the cards did not change within 5s". maxWait is 5 seconds.
  • TestStreamNotDroppedWhileLinesArrive, internal/streamer/streamer_internal_test.go: a 100 ms idle timeout, so any stall over 100 ms ends the stream and fails the test.

Change.

  1. In home_internal_test.go, maxWait becomes 15 seconds, the value stepTimeout has in internal/routewatch since the fix above. The wait returns as soon as the condition holds, so a passing run takes no longer.
  2. TestStreamNotDroppedWhileLinesArrive must fail only when a line does not restart the idle timer. A stall of several seconds on the host must not fail it, and it may add at most a few seconds to the suite. It still has to show that lines arriving for longer than the idle timeout keep the stream up.
  3. Any other test in the repository that fails when a fixed wall-clock window shorter than a few seconds runs out, on a host that is merely slow, gets the same treatment. Say on the PR which you changed.

No production code changes, and no timeout goes in the Makefile, Dockerfile or script/.

Done when. script/cibuild passes on the branch while the host is busy, and the three tests are shown to still fail when the code they guard is broken (one line on the PR saying how).

Model: opus-5-5

`script/cibuild` on `next` at `e58c663` failed in the test step on a host with a load average of about 80, in the tests below. One failure followed a stall of about 5.5 seconds, visible in the same log as HTTP requests that took that long. The same commit passed on another run under the same load, so the failure is intermittent. A check that a busy host can fail is a defect in the tests: the code was not wrong. https://git.eeqj.de/sneak/routewatch/issues/213 fixed the same flaw in the routewatch package. **Failing.** - `TestKeepHomeCardsFreshWorksThemOutEveryInterval` and `TestStartedServerWorksOutTheCards`, `internal/server/home_internal_test.go`: "the cards did not change within 5s". `maxWait` is 5 seconds. - `TestStreamNotDroppedWhileLinesArrive`, `internal/streamer/streamer_internal_test.go`: a 100 ms idle timeout, so any stall over 100 ms ends the stream and fails the test. **Change.** 1. In `home_internal_test.go`, `maxWait` becomes 15 seconds, the value `stepTimeout` has in `internal/routewatch` since the fix above. The wait returns as soon as the condition holds, so a passing run takes no longer. 2. `TestStreamNotDroppedWhileLinesArrive` must fail only when a line does not restart the idle timer. A stall of several seconds on the host must not fail it, and it may add at most a few seconds to the suite. It still has to show that lines arriving for longer than the idle timeout keep the stream up. 3. Any other test in the repository that fails when a fixed wall-clock window shorter than a few seconds runs out, on a host that is merely slow, gets the same treatment. Say on the PR which you changed. No production code changes, and no timeout goes in the Makefile, `Dockerfile` or `script/`. **Done when.** `script/cibuild` passes on the branch while the host is busy, and the three tests are shown to still fail when the code they guard is broken (one line on the PR saying how). Model: opus-5-5
clawbot self-assigned this 2026-11-10 10:19:18 +01:00
Author
Collaborator

Built in #242, test code only. The pull request lists each test changed.

Model: opus-5-5

Built in https://git.eeqj.de/sneak/routewatch/pulls/242, test code only. The pull request lists each test changed. Model: opus-5-5
Author
Collaborator

script/cibuild failed the same way on next at 28ef027: TestKeepHomeCardsFreshWorksThemOutEveryInterval and TestStartedServerWorksOutTheCards, "the cards did not change within 5s", with requests in the same log taking up to 5.8 seconds. Nothing else merges to next until the fix in #242 has landed.

Model: opus-5-5

`script/cibuild` failed the same way on `next` at `28ef027`: `TestKeepHomeCardsFreshWorksThemOutEveryInterval` and `TestStartedServerWorksOutTheCards`, "the cards did not change within 5s", with requests in the same log taking up to 5.8 seconds. Nothing else merges to `next` until the fix in https://git.eeqj.de/sneak/routewatch/pulls/242 has landed. Model: opus-5-5
clawbot added the critical label 2026-11-10 12:17:01 +01:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/routewatch#240