Show a target paused by its circuit breaker (closes #385)
check / check (push) Successful in 3m20s
check / check (push) Successful in 3m20s
While an http or slack target's circuit breaker is open, the target's row on the webhook page says its deliveries are paused until the cooldown ends, then one is sent to test the target. Each of its retrying deliveries shows as waiting in the event log and on the event's page, with the earliest it can be tried next: the later of the cooldown's end and the end of its own backoff, dated when not today in UTC. While the breaker is half-open, the row says deliveries are held while one delivery tests the target, with no time, and deliveries keep their plain status. The engine gains one read, StateAndCooldown(targetID), taking a breaker's state and remaining cooldown under one lock; the handlers reach it through a one-method interface wired like Archives. Model: opus-5-5
This commit is contained in:
@@ -234,7 +234,7 @@ func (c *httpCore) handleRetry(
|
||||
database.DeliveryStatusRetrying,
|
||||
)
|
||||
|
||||
backoff := calcBackoff(attemptNum)
|
||||
backoff := Backoff(attemptNum)
|
||||
|
||||
retryTask := *task
|
||||
retryTask.AttemptNum = attemptNum + 1
|
||||
@@ -301,7 +301,7 @@ func (c *httpCore) remainingBackoff(
|
||||
return 0
|
||||
}
|
||||
|
||||
backoff := calcBackoff(attemptNum)
|
||||
backoff := Backoff(attemptNum)
|
||||
elapsed := time.Since(lastResult.CreatedAt)
|
||||
remaining := backoff - elapsed
|
||||
|
||||
@@ -326,12 +326,14 @@ func (c *httpCore) backoffElapsed(
|
||||
return true
|
||||
}
|
||||
|
||||
backoff := calcBackoff(attemptNum)
|
||||
backoff := Backoff(attemptNum)
|
||||
|
||||
return time.Since(lastResult.CreatedAt) >= backoff
|
||||
}
|
||||
|
||||
func calcBackoff(attemptNum int) time.Duration {
|
||||
// Backoff is how long an http or slack target with retries waits after
|
||||
// a delivery's failed attempt attemptNum before trying it again.
|
||||
func Backoff(attemptNum int) time.Duration {
|
||||
shift := max(attemptNum-1, 0)
|
||||
shift = min(shift, maxBackoffShift)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user