Show a target paused by its circuit breaker (closes #385)
check / check (push) Successful in 3m21s
check / check (push) Successful in 3m21s
While an http or slack target's circuit breaker turns its deliveries away, the target's row on the webhook page says its deliveries are paused and until when, in UTC and from now. Each of its retrying deliveries shows as waiting, with the reason and the same time, in the event log and on the event's page, instead of a bare "retrying". The engine gains one read, CooldownRemaining(targetID), over the breakers it already keeps; the handlers reach it through a one-method interface wired like Archives. Model: opus-5-5
This commit is contained in:
@@ -143,6 +143,15 @@ type Archives interface {
|
||||
Rename(targetID, webhookName, targetName string) error
|
||||
}
|
||||
|
||||
// CircuitBreakers is how the handlers read a target's circuit
|
||||
// breaker, so the webhook page and the event log can say that
|
||||
// deliveries to the target are paused and until when. Like Archives,
|
||||
// it keeps the handlers free of the engine's internals and is
|
||||
// trivially faked in tests.
|
||||
type CircuitBreakers interface {
|
||||
CooldownRemaining(targetID string) time.Duration
|
||||
}
|
||||
|
||||
// EngineParams are the fx dependencies for the delivery
|
||||
// engine.
|
||||
type EngineParams struct {
|
||||
@@ -186,9 +195,11 @@ type Engine struct {
|
||||
// targets maps each target type to its implementation.
|
||||
targets map[database.TargetType]Target
|
||||
|
||||
// httpTarget is retained so tests can reach the HTTP
|
||||
// target's shared client and circuit breakers.
|
||||
httpTarget *httpTarget
|
||||
// httpTarget and slackTarget are retained so CooldownRemaining
|
||||
// can read their circuit breakers, and so tests can reach the
|
||||
// HTTP target's shared client.
|
||||
httpTarget *httpTarget
|
||||
slackTarget *slackTarget
|
||||
|
||||
// dbTarget is retained so the engine can reach the archive
|
||||
// writer registry for eviction, renames and the idle sweep.
|
||||
@@ -300,6 +311,26 @@ func (e *Engine) Rename(
|
||||
return e.dbTarget.rename(targetID, webhookName, targetName)
|
||||
}
|
||||
|
||||
// CooldownRemaining implements CircuitBreakers. It is
|
||||
// CircuitBreaker.CooldownRemaining for the target's breaker: how long
|
||||
// the breaker will keep turning the target's deliveries away, and
|
||||
// zero when it lets them through. A target with no breaker gets zero,
|
||||
// and reading never creates one.
|
||||
func (e *Engine) CooldownRemaining(targetID string) time.Duration {
|
||||
for _, core := range []*httpCore{
|
||||
e.httpTarget.httpCore, e.slackTarget.httpCore,
|
||||
} {
|
||||
val, ok := core.circuitBreakers.Load(targetID)
|
||||
if ok {
|
||||
cb, _ := val.(*CircuitBreaker)
|
||||
|
||||
return cb.CooldownRemaining()
|
||||
}
|
||||
}
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
// ScheduleRetry schedules a task to be re-enqueued onto the
|
||||
// retry channel after delay. It implements the Scheduler
|
||||
// interface the targets use to own their durable retries.
|
||||
|
||||
Reference in New Issue
Block a user