feat: add retry with exponential backoff for notification delivery
All checks were successful
check / check (push) Successful in 42s

Notifications were fire-and-forget: if Slack, Mattermost, or ntfy was
temporarily down, changes were silently lost. This adds automatic retry
with exponential backoff and jitter to all notification endpoints.

Implementation:
- New retry.go with configurable RetryConfig (max retries, base delay,
  max delay) and exponential backoff with ±25% jitter
- Each dispatch goroutine now wraps its send call in deliverWithRetry
- Default: 3 retries (4 total attempts), 1s base delay, 10s max delay
- Context-aware: respects cancellation during retry sleep
- Structured logging on each retry attempt and on final success after
  retry

All existing tests continue to pass. New tests cover:
- Backoff calculation (increase, cap)
- Retry success on first attempt (no unnecessary retries)
- Retry on transient failure (succeeds after N attempts)
- Exhausted retries (returns last error)
- Context cancellation during retry sleep
- Integration: SendNotification retries transient 500s
- Integration: all three endpoints retry independently
- Integration: permanent failure exhausts retries

closes #62
This commit is contained in:
clawbot
2026-03-10 11:11:32 -07:00
parent b64db3e10f
commit 31bd6c3228
4 changed files with 693 additions and 12 deletions

View File

@@ -6,6 +6,7 @@ import (
"log/slog"
"net/http"
"net/url"
"time"
)
// NtfyPriority exports ntfyPriority for testing.
@@ -74,3 +75,31 @@ func (svc *Service) SendSlack(
ctx, webhookURL, title, message, priority,
)
}
// SetRetryConfig overrides the retry configuration for
// testing.
func (svc *Service) SetRetryConfig(cfg RetryConfig) {
svc.retryConfig = cfg
}
// SetSleepFunc overrides the sleep function so tests can
// eliminate real delays.
func (svc *Service) SetSleepFunc(
fn func(time.Duration) <-chan time.Time,
) {
svc.sleepFn = fn
}
// DeliverWithRetry exports deliverWithRetry for testing.
func (svc *Service) DeliverWithRetry(
ctx context.Context,
endpoint string,
fn func(context.Context) error,
) error {
return svc.deliverWithRetry(ctx, endpoint, fn)
}
// BackoffDuration exports RetryConfig.backoff for testing.
func (rc RetryConfig) BackoffDuration(attempt int) time.Duration {
return rc.defaults().backoff(attempt)
}