Circuit breaker half-open state busy-loops queued tasks with a DB write per spin #306

Closed
opened 2026-08-30 03:57:20 +02:00 by clawbot · 2 comments
Collaborator

From audit #303 (significant).

In half-open, Allow() returns false while CooldownRemaining() returns 0 (internal/delivery/circuit_breaker.go:71-73, :85-87), so circuitBreakerBlock reschedules with delay 0 (internal/delivery/target_http.go:145-165): every queued task for the target spins through retryCh with a DB status write per iteration for the whole probe window (up to 30s).

Done: blocked tasks wait a non-zero delay (or until the probe resolves); no per-spin DB writes; test covers half-open with queued tasks.

From audit #303 (significant). In half-open, `Allow()` returns false while `CooldownRemaining()` returns 0 (`internal/delivery/circuit_breaker.go:71-73`, `:85-87`), so `circuitBreakerBlock` reschedules with delay 0 (`internal/delivery/target_http.go:145-165`): every queued task for the target spins through `retryCh` with a DB status write per iteration for the whole probe window (up to 30s). Done: blocked tasks wait a non-zero delay (or until the probe resolves); no per-spin DB writes; test covers half-open with queued tasks.
clawbot added this to the 1.0.0 milestone 2026-09-21 09:20:32 +02:00
Author
Collaborator

Plan. The loop is still on next (3cdab97). In half-open, Allow() returns false and CooldownRemaining() returns 0 (internal/delivery/circuit_breaker.go). So circuitBreakerBlock (internal/delivery/target_http.go) reschedules every blocked task at once, and writes its status each time.

  • Delay: a task blocked by a half-open breaker waits a non-zero delay. The plainest is the breaker's cooldown, which also bounds how long the one probe is allowed before the breaker reopens. If the probe succeeds early, a blocked task still waits out the rest of its delay; that is acceptable. The delay may be shorter than the cooldown, but it must never be zero.
  • Naming: the breaker needs one clearly named method, or a clear change to CooldownRemaining, for how long a blocked task should wait in each state. Its comment says what it returns in closed, open and half-open. Do not coin a new term for the half-open wait.
  • Writes: a blocked task whose delivery already says retrying does not write the status again, so no DB write happens per reschedule. The first block still sets retrying.
  • Tests:
    • With the breaker half-open and tasks queued, each blocked task is rescheduled with a non-zero delay, and the status is written once, not once per pass.
    • The closed and open behaviour is unchanged.
    • Each test fails when its part of the fix is removed.
  • Unchanged: max-retries accounting, the gauge publishing, and the probe itself.

Model: opus-5-5

Plan. The loop is still on `next` (`3cdab97`). In half-open, `Allow()` returns false and `CooldownRemaining()` returns 0 (`internal/delivery/circuit_breaker.go`). So `circuitBreakerBlock` (`internal/delivery/target_http.go`) reschedules every blocked task at once, and writes its status each time. - **Delay:** a task blocked by a half-open breaker waits a non-zero delay. The plainest is the breaker's cooldown, which also bounds how long the one probe is allowed before the breaker reopens. If the probe succeeds early, a blocked task still waits out the rest of its delay; that is acceptable. The delay may be shorter than the cooldown, but it must never be zero. - **Naming:** the breaker needs one clearly named method, or a clear change to `CooldownRemaining`, for how long a blocked task should wait in each state. Its comment says what it returns in closed, open and half-open. Do not coin a new term for the half-open wait. - **Writes:** a blocked task whose delivery already says `retrying` does not write the status again, so no DB write happens per reschedule. The first block still sets `retrying`. - **Tests:** - With the breaker half-open and tasks queued, each blocked task is rescheduled with a non-zero delay, and the status is written once, not once per pass. - The closed and open behaviour is unchanged. - Each test fails when its part of the fix is removed. - **Unchanged:** max-retries accounting, the gauge publishing, and the probe itself. Model: opus-5-5
clawbot self-assigned this 2026-09-29 04:31:40 +02:00
Author
Collaborator

Implemented in #330: a delivery refused while the breaker is half-open now waits a whole cooldown, and a refused delivery already at retrying is not written again. One side effect, disclosed on the PR: webhooker_delivery_retries_total no longer counts a refusal of a delivery that is already retrying.

Model: opus-5-5

Implemented in https://git.eeqj.de/sneak/webhooker/pulls/330: a delivery refused while the breaker is half-open now waits a whole cooldown, and a refused delivery already at `retrying` is not written again. One side effect, disclosed on the PR: `webhooker_delivery_retries_total` no longer counts a refusal of a delivery that is already `retrying`. Model: opus-5-5
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/webhooker#306