Show a target paused by its circuit breaker (closes #385)
check / check (push) Successful in 3m16s
check / check (push) Successful in 3m16s
A target whose circuit breaker had tripped still showed as Active, and its deliveries sat at a bare "retrying" with no attempts. Its row on the webhook page now says deliveries are paused after repeated failures and until when the cooldown ends, adding that one waiting delivery is then sent to test the target; while half-open it says deliveries are held while one tests it, with no time. Waiting deliveries show "next try no earlier than" the later of the cooldown and their own backoff, with the date when not today. The engine gains one read of a breaker's state and remaining cooldown under one lock, and shares the backoff formula. Model: opus-5-5
This commit was merged in pull request #478.
This commit is contained in:
@@ -143,6 +143,15 @@ type Archives interface {
|
||||
Rename(targetID, webhookName, targetName string) error
|
||||
}
|
||||
|
||||
// CircuitBreakers is how the handlers read a target's circuit
|
||||
// breaker, so the webhook page and the event log can say that
|
||||
// deliveries to the target are paused and until when. Like Archives,
|
||||
// it keeps the handlers free of the engine's internals and is
|
||||
// trivially faked in tests.
|
||||
type CircuitBreakers interface {
|
||||
StateAndCooldown(targetID string) (CircuitState, time.Duration)
|
||||
}
|
||||
|
||||
// EngineParams are the fx dependencies for the delivery
|
||||
// engine.
|
||||
type EngineParams struct {
|
||||
@@ -186,9 +195,11 @@ type Engine struct {
|
||||
// targets maps each target type to its implementation.
|
||||
targets map[database.TargetType]Target
|
||||
|
||||
// httpTarget is retained so tests can reach the HTTP
|
||||
// target's shared client and circuit breakers.
|
||||
httpTarget *httpTarget
|
||||
// httpTarget and slackTarget are retained so StateAndCooldown
|
||||
// can read their circuit breakers, and so tests can reach the
|
||||
// HTTP target's shared client.
|
||||
httpTarget *httpTarget
|
||||
slackTarget *slackTarget
|
||||
|
||||
// dbTarget is retained so the engine can reach the archive
|
||||
// writer registry for eviction, renames and the idle sweep.
|
||||
@@ -300,6 +311,28 @@ func (e *Engine) Rename(
|
||||
return e.dbTarget.rename(targetID, webhookName, targetName)
|
||||
}
|
||||
|
||||
// StateAndCooldown implements CircuitBreakers. It returns the state of
|
||||
// the target's circuit breaker and, while the breaker is open, what is
|
||||
// left of its cooldown; the cooldown is zero once that has passed and
|
||||
// in any other state. A target with no breaker reads as closed with no
|
||||
// cooldown, and reading never creates one.
|
||||
func (e *Engine) StateAndCooldown(
|
||||
targetID string,
|
||||
) (CircuitState, time.Duration) {
|
||||
for _, core := range []*httpCore{
|
||||
e.httpTarget.httpCore, e.slackTarget.httpCore,
|
||||
} {
|
||||
val, ok := core.circuitBreakers.Load(targetID)
|
||||
if ok {
|
||||
cb, _ := val.(*CircuitBreaker)
|
||||
|
||||
return cb.StateAndCooldown()
|
||||
}
|
||||
}
|
||||
|
||||
return CircuitClosed, 0
|
||||
}
|
||||
|
||||
// ScheduleRetry schedules a task to be re-enqueued onto the
|
||||
// retry channel after delay. It implements the Scheduler
|
||||
// interface the targets use to own their durable retries.
|
||||
|
||||
Reference in New Issue
Block a user