Use one recovery routine at start and on every sweep tick (closes #542)
check / check (push) Waiting to run
check / check (push) Waiting to run
Start-up recovery and the 60-second sweep were two parallel sets of functions doing the same job. recoverDeliveries now does it, run once at start and then on every tick from one goroutine. A retrying delivery is scheduled after what is left of its backoff, now if nothing is left, so backoffElapsed is gone. The run at start still takes every pending delivery and opens a webhook database whose file is missing; a tick still takes only pending deliveries untouched for 15 minutes, at most 500, and skips a missing file. A retrying delivery is taken before its event is read. The README describes one routine. Model: opus-5-5
This commit was merged in pull request #553.
This commit is contained in:
@@ -84,8 +84,8 @@ func (c *httpCore) deliver(
|
||||
// engine is stopping, which cancels the context its request is made
|
||||
// with. Such an attempt is not recorded: no DeliveryResult, no circuit
|
||||
// breaker outcome, no status change, no count in the attempt metrics.
|
||||
// The delivery stays pending or retrying, and startup recovery sends
|
||||
// it again on the next start.
|
||||
// The delivery stays pending or retrying, and the recovery routine
|
||||
// sends it again on the next start.
|
||||
//
|
||||
// A target's own timeout leaves that context alone, so its attempt is
|
||||
// still recorded as the failure it is.
|
||||
@@ -334,7 +334,8 @@ func (c *httpCore) publishCircuitState(
|
||||
|
||||
// remainingBackoff returns how long remains of the backoff
|
||||
// window for the last attempt of a recovered retrying
|
||||
// delivery. It implements rescheduler.
|
||||
// delivery: zero once it has passed, or when the last attempt
|
||||
// cannot be read. It implements rescheduler.
|
||||
func (c *httpCore) remainingBackoff(
|
||||
webhookDB *gorm.DB,
|
||||
deliveryID string,
|
||||
@@ -357,29 +358,6 @@ func (c *httpCore) remainingBackoff(
|
||||
return max(remaining, 0)
|
||||
}
|
||||
|
||||
// backoffElapsed reports whether the backoff window for the
|
||||
// last attempt of a retrying delivery has passed. It
|
||||
// implements rescheduler.
|
||||
func (c *httpCore) backoffElapsed(
|
||||
webhookDB *gorm.DB,
|
||||
deliveryID string,
|
||||
attemptNum int,
|
||||
) bool {
|
||||
var lastResult database.DeliveryResult
|
||||
|
||||
err := webhookDB.
|
||||
Where("delivery_id = ?", deliveryID).
|
||||
Order("created_at DESC").
|
||||
First(&lastResult).Error
|
||||
if err != nil {
|
||||
return true
|
||||
}
|
||||
|
||||
backoff := Backoff(attemptNum)
|
||||
|
||||
return time.Since(lastResult.CreatedAt) >= backoff
|
||||
}
|
||||
|
||||
// Backoff is how long an http or slack target with retries waits after
|
||||
// a delivery's failed attempt attemptNum before trying it again.
|
||||
func Backoff(attemptNum int) time.Duration {
|
||||
|
||||
Reference in New Issue
Block a user