Make deliveries refused while half-open wait a cooldown (closes #306)
check / check (push) Successful in 4m59s
check / check (push) Successful in 4m59s
While the breaker was half-open, Allow refused every delivery but the probe and CooldownRemaining returned zero, so each queued task for the target went straight back onto the retry channel and rewrote its status on every pass until the probe finished. CooldownRemaining now returns the whole cooldown while half-open, so a refused delivery waits that long. A refused delivery already at retrying is not written again, so the retry counter now moves only when a refusal moves a delivery into retrying. Model: opus-5-5
This commit is contained in:
@@ -2051,7 +2051,7 @@ rescans the database anyway).
|
||||
| ----------- | -------- |
|
||||
| **Closed** | Normal operation. Deliveries flow through. Consecutive failures are counted. |
|
||||
| **Open** | Target appears down. Deliveries are skipped and rescheduled for after the cooldown. |
|
||||
| **Half-Open** | Cooldown expired. One probe delivery is allowed to test if the target has recovered. |
|
||||
| **Half-Open** | Cooldown expired. One probe delivery is allowed to test if the target has recovered. Other deliveries are rescheduled for one whole cooldown later. |
|
||||
|
||||
**Transitions:**
|
||||
|
||||
@@ -2089,7 +2089,9 @@ operations), and log targets (stdout) do not use circuit breakers.
|
||||
When a circuit is open and a new delivery arrives, the engine marks the
|
||||
delivery as `retrying` and schedules a retry timer for after the
|
||||
remaining cooldown period. This ensures no deliveries are lost — they're
|
||||
just delayed until the target is healthy again.
|
||||
just delayed until the target is healthy again. A delivery already in
|
||||
`retrying` keeps that status without another database write each time
|
||||
the breaker turns it away.
|
||||
|
||||
### Metrics
|
||||
|
||||
@@ -2103,7 +2105,7 @@ arriving and being stored, they are just not getting anywhere.
|
||||
| Metric | Type | Meaning |
|
||||
| ------ | ---- | ------- |
|
||||
| `webhooker_events_received_total` | counter | Events received and durably stored. Compare against the delivery counters on one dashboard |
|
||||
| `webhooker_delivery_attempts_total` | counter | Delivery attempts actually dispatched to a target. A delivery an open circuit breaker refused is not one: it is counted as a retry instead |
|
||||
| `webhooker_delivery_attempts_total` | counter | Delivery attempts actually dispatched to a target. A delivery a circuit breaker refused is not one: it is counted as a retry instead, but only when the refusal moves it into `retrying` |
|
||||
| `webhooker_deliveries_succeeded_total` | counter | Deliveries that reached `delivered` |
|
||||
| `webhooker_deliveries_failed_total` | counter | Deliveries that failed terminally and will not be retried |
|
||||
| `webhooker_delivery_retries_total` | counter | Deliveries put back into `retrying` |
|
||||
|
||||
Reference in New Issue
Block a user