Show a target paused by its circuit breaker (closes #385)
check / check (push) Successful in 3m21s

While an http or slack target's circuit breaker is open, the target's
row on the webhook page says its deliveries are paused until the
cooldown ends, in UTC and from now. Each of its retrying deliveries
shows as waiting in the event log and on the event's page, until the
later of the cooldown's end and the end of its own backoff. While the
breaker is half-open, the row says deliveries are held while one
delivery tests the target, with no time, and deliveries keep their
plain status.

The engine gains one read, StateAndCooldown(targetID), taking a
breaker's state and remaining cooldown under one lock; the handlers
reach it through a one-method interface wired like Archives.

Model: opus-5-5
This commit is contained in:
2026-10-02 23:16:44 +00:00
parent 3489d6909a
commit d66292df43
19 changed files with 520 additions and 28 deletions
+41
View File
@@ -9,6 +9,7 @@ import (
"net/http/httptest"
"sync"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -181,6 +182,40 @@ func (r *recordingArchives) Renames() []archiveRename {
return out
}
// testCircuitBreakers is a delivery.CircuitBreakers that reports, for
// each target, the circuit state and cooldown a test gave it with Set,
// and a closed breaker for any other target.
type testCircuitBreakers struct {
mu sync.Mutex
states map[string]delivery.CircuitState
cooldowns map[string]time.Duration
}
// Set makes the target's breaker read as state, with cooldown left.
func (b *testCircuitBreakers) Set(
targetID string, state delivery.CircuitState, cooldown time.Duration,
) {
b.mu.Lock()
defer b.mu.Unlock()
if b.states == nil {
b.states = map[string]delivery.CircuitState{}
b.cooldowns = map[string]time.Duration{}
}
b.states[targetID] = state
b.cooldowns[targetID] = cooldown
}
func (b *testCircuitBreakers) StateAndCooldown(
targetID string,
) (delivery.CircuitState, time.Duration) {
b.mu.Lock()
defer b.mu.Unlock()
return b.states[targetID], b.cooldowns[targetID]
}
// newTestApp returns an app whose RequireStart fails the test when
// starting takes longer than fx's default start timeout of 15s. That
// limit catches a start that hangs, not a busy host: measured with make
@@ -231,6 +266,12 @@ func newTestAppWithConfig(
func(r *recordingArchives) delivery.Archives {
return r
},
func() *testCircuitBreakers {
return &testCircuitBreakers{}
},
func(b *testCircuitBreakers) delivery.CircuitBreakers {
return b
},
metrics.NewRegistry,
metrics.New,
middleware.New,