Make deliveries refused while half-open wait a cooldown (closes #306) #330

Merged
clawbot merged 1 commits from issue-306-half-open-busy-loop into next 2026-09-29 05:48:21 +02:00
Collaborator

Fixes #306.

While a target's circuit breaker was half-open, Allow refused every delivery except the probe, but CooldownRemaining returned zero. So every queued task for that target went straight back onto the retry channel, and each pass rewrote the delivery's status, for as long as the probe took.

  • CooldownRemaining now returns the whole cooldown while half-open, so a delivery refused then waits that long. If the probe fails, the circuit reopens for exactly that long. If the probe succeeds early, refused deliveries still wait out their delay. Closed and open return what they did before; the comment now says what each state returns.
  • circuitBreakerBlock no longer rewrites the status of a delivery that is already retrying. The first refusal of a pending delivery still sets retrying.
  • README: the half-open row, the paragraph under the breaker diagram and the webhooker_delivery_attempts_total row now describe this.

A new engine test puts the breaker into half-open and sends several queued deliveries through it more than once each. It checks that every retry is given the cooldown as its delay, and that each delivery's status is written once.

Disclosures:

  • Behaviour change: webhooker_delivery_retries_total moves only with a status write, so it no longer counts a breaker refusing a delivery that is already retrying. TestDeliveryMetrics_BreakerBlockedIsNotAnAttempt now expects that.
  • Test-only helpers added to export_test.go: ExportSetCircuitBreaker and ExportDeliverHTTPWithScheduler.

Model: opus-5-5

Fixes https://git.eeqj.de/sneak/webhooker/issues/306. While a target's circuit breaker was half-open, `Allow` refused every delivery except the probe, but `CooldownRemaining` returned zero. So every queued task for that target went straight back onto the retry channel, and each pass rewrote the delivery's status, for as long as the probe took. - `CooldownRemaining` now returns the whole cooldown while half-open, so a delivery refused then waits that long. If the probe fails, the circuit reopens for exactly that long. If the probe succeeds early, refused deliveries still wait out their delay. Closed and open return what they did before; the comment now says what each state returns. - `circuitBreakerBlock` no longer rewrites the status of a delivery that is already `retrying`. The first refusal of a `pending` delivery still sets `retrying`. - README: the half-open row, the paragraph under the breaker diagram and the `webhooker_delivery_attempts_total` row now describe this. A new engine test puts the breaker into half-open and sends several queued deliveries through it more than once each. It checks that every retry is given the cooldown as its delay, and that each delivery's status is written once. Disclosures: - Behaviour change: `webhooker_delivery_retries_total` moves only with a status write, so it no longer counts a breaker refusing a delivery that is already `retrying`. `TestDeliveryMetrics_BreakerBlockedIsNotAnAttempt` now expects that. - Test-only helpers added to `export_test.go`: `ExportSetCircuitBreaker` and `ExportDeliverHTTPWithScheduler`. Model: opus-5-5
clawbot added the needs-review label 2026-09-29 05:08:08 +02:00
clawbot self-assigned this 2026-09-29 05:08:09 +02:00
clawbot added 1 commit 2026-09-29 05:08:09 +02:00
While the breaker was half-open, Allow refused every delivery but the
probe and CooldownRemaining returned zero, so each queued task for the
target went straight back onto the retry channel and rewrote its status
on every pass until the probe finished.

CooldownRemaining now returns the whole cooldown while half-open, so a
refused delivery waits that long. A refused delivery already at
retrying is not written again, so the retry counter now moves only
when a refusal moves a delivery into retrying.

Model: opus-5-5
Author
Collaborator

Review passed.

Model: opus-5-5

Review passed. Model: opus-5-5
clawbot merged commit e0b211f960 into next 2026-09-29 05:48:21 +02:00
clawbot deleted branch issue-306-half-open-busy-loop 2026-09-29 05:48:22 +02:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/webhooker#330