feat: add retry with exponential backoff for notification delivery
All checks were successful
check / check (push) Successful in 42s
All checks were successful
check / check (push) Successful in 42s
Notifications were fire-and-forget: if Slack, Mattermost, or ntfy was temporarily down, changes were silently lost. This adds automatic retry with exponential backoff and jitter to all notification endpoints. Implementation: - New retry.go with configurable RetryConfig (max retries, base delay, max delay) and exponential backoff with ±25% jitter - Each dispatch goroutine now wraps its send call in deliverWithRetry - Default: 3 retries (4 total attempts), 1s base delay, 10s max delay - Context-aware: respects cancellation during retry sleep - Structured logging on each retry attempt and on final success after retry All existing tests continue to pass. New tests cover: - Backoff calculation (increase, cap) - Retry success on first attempt (no unnecessary retries) - Retry on transient failure (succeeds after N attempts) - Exhausted retries (returns last error) - Context cancellation during retry sleep - Integration: SendNotification retries transient 500s - Integration: all three endpoints retry independently - Integration: permanent failure exhausts retries closes #62
This commit is contained in:
@@ -6,6 +6,7 @@ import (
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"time"
|
||||
)
|
||||
|
||||
// NtfyPriority exports ntfyPriority for testing.
|
||||
@@ -74,3 +75,31 @@ func (svc *Service) SendSlack(
|
||||
ctx, webhookURL, title, message, priority,
|
||||
)
|
||||
}
|
||||
|
||||
// SetRetryConfig overrides the retry configuration for
|
||||
// testing.
|
||||
func (svc *Service) SetRetryConfig(cfg RetryConfig) {
|
||||
svc.retryConfig = cfg
|
||||
}
|
||||
|
||||
// SetSleepFunc overrides the sleep function so tests can
|
||||
// eliminate real delays.
|
||||
func (svc *Service) SetSleepFunc(
|
||||
fn func(time.Duration) <-chan time.Time,
|
||||
) {
|
||||
svc.sleepFn = fn
|
||||
}
|
||||
|
||||
// DeliverWithRetry exports deliverWithRetry for testing.
|
||||
func (svc *Service) DeliverWithRetry(
|
||||
ctx context.Context,
|
||||
endpoint string,
|
||||
fn func(context.Context) error,
|
||||
) error {
|
||||
return svc.deliverWithRetry(ctx, endpoint, fn)
|
||||
}
|
||||
|
||||
// BackoffDuration exports RetryConfig.backoff for testing.
|
||||
func (rc RetryConfig) BackoffDuration(attempt int) time.Duration {
|
||||
return rc.defaults().backoff(attempt)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user