notify: drain in-flight deliveries at shutdown (closes #106)
All checks were successful
check / check (push) Successful in 33s
All checks were successful
check / check (push) Successful in 33s
notify.New accepted an fx.Lifecycle and never used it, so the three dispatch goroutines were untracked. context.WithoutCancel kept a delivery alive past its caller's cancellation but made nothing wait for it: the process could exit while a delivery was still in its retry backoff (up to five attempts, 60s max delay), silently losing exactly the alert most worth keeping. Deliveries are now tracked in a sync.WaitGroup whose counter is incremented on the dispatching goroutine before the worker starts, and notify.New registers an OnStop hook that drains them. The drain is bounded by the context fx passes to OnStop; when it expires with work outstanding, the count is logged at warn level and parked retry backoffs are released via an abandon channel so they stop retrying rather than outliving the drain. Deliveries submitted after the drain has begun are refused and logged, so a stream of new notifications cannot extend shutdown indefinitely. The three near-identical dispatchers now share one tracked dispatch helper. Tests use httptest servers and the existing retry knobs (SetRetryConfig/SetSleepFunc) so nothing waits on a real backoff. README's shutdown claim is reworded to match the bounded semantics.
This commit is contained in:
13
README.md
13
README.md
@@ -218,7 +218,8 @@ internal/
|
||||
- **Structured logging**: All logs use `log/slog` with JSON output in
|
||||
production (TTY detection for development).
|
||||
- **Graceful shutdown**: All background goroutines respect context
|
||||
cancellation and the fx lifecycle.
|
||||
cancellation and the fx lifecycle. In-flight notification deliveries
|
||||
are drained on shutdown, bounded by the shutdown timeout.
|
||||
|
||||
---
|
||||
|
||||
@@ -452,8 +453,14 @@ docker run -d \
|
||||
from a previous cycle.
|
||||
4. **On change detection**: Send notifications to all configured
|
||||
endpoints, update in-memory state, persist to disk.
|
||||
5. **Shutdown**: Persist final state to disk, complete in-flight
|
||||
notifications, stop gracefully.
|
||||
5. **Shutdown**: Persist final state to disk, wait for in-flight
|
||||
notification deliveries to complete, stop gracefully. The wait is
|
||||
bounded by the fx shutdown timeout (15s by default): deliveries still
|
||||
retrying against an unreachable endpoint when that expires are
|
||||
abandoned, and the number abandoned is logged at warn level rather
|
||||
than dropped silently. Notifications generated after shutdown has
|
||||
begun are refused and logged, so a late burst cannot extend the
|
||||
shutdown.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user