feat: parallel fan-out delivery + circuit breaker for retry targets
All checks were successful
check / check (push) Successful in 1m52s
All checks were successful
check / check (push) Successful in 1m52s
- Fan out all targets for an event in parallel goroutines (fire-and-forget) - Add per-target circuit breaker for retry targets (closed/open/half-open) - Circuit breaker trips after 5 consecutive failures, 30s cooldown - Open circuit skips delivery and reschedules after cooldown - Half-open allows one probe delivery to test recovery - HTTP/database/log targets unaffected (no circuit breaker) - Recovery path also fans out in parallel - Update README with parallel delivery and circuit breaker docs
This commit is contained in:
95
README.md
95
README.md
@@ -498,14 +498,89 @@ External Service
|
||||
│ Engine │ (backoff)
|
||||
└──────┬───────┘
|
||||
│
|
||||
┌────────────────────┼────────────────────┐
|
||||
▼ ▼ ▼
|
||||
┌────────────┐ ┌────────────┐ ┌────────────┐
|
||||
│ HTTP Target│ │Retry Target│ │ Log Target │
|
||||
│ (1 attempt)│ │ (backoff) │ │ (stdout) │
|
||||
└────────────┘ └────────────┘ └────────────┘
|
||||
┌─── parallel goroutines (fan-out) ───┐
|
||||
▼ ▼ ▼
|
||||
┌────────────┐ ┌────────────┐ ┌────────────┐
|
||||
│ HTTP Target│ │Retry Target│ │ Log Target │
|
||||
│ (1 attempt)│ │ (backoff + │ │ (stdout) │
|
||||
└────────────┘ │ circuit │ └────────────┘
|
||||
│ breaker) │
|
||||
└────────────┘
|
||||
```
|
||||
|
||||
### Parallel Fan-Out Delivery
|
||||
|
||||
When the delivery engine receives a batch of tasks for an event, it
|
||||
fans out **all targets in parallel** — each `DeliveryTask` is dispatched
|
||||
in its own goroutine immediately. An HTTP target, a retry target, and
|
||||
a log target for the same event all start delivering simultaneously
|
||||
with no sequential bottleneck.
|
||||
|
||||
This means:
|
||||
|
||||
- **No head-of-line blocking** — a slow HTTP target doesn't delay the
|
||||
log target or other targets.
|
||||
- **Maximum throughput** — all targets receive the event as quickly as
|
||||
possible.
|
||||
- **Independent results** — each goroutine records its own delivery
|
||||
result in the per-webhook database without coordination.
|
||||
- **Fire-and-forget** — the engine doesn't wait for all goroutines to
|
||||
finish; each delivery is completely independent.
|
||||
|
||||
The same parallel fan-out applies to crash recovery: when the engine
|
||||
restarts and finds pending deliveries in per-webhook databases, it
|
||||
recovers them and fans them out in parallel just like fresh deliveries.
|
||||
|
||||
### Circuit Breaker (Retry Targets)
|
||||
|
||||
Retry targets are protected by a **per-target circuit breaker** that
|
||||
prevents hammering a down target with repeated failed delivery attempts.
|
||||
The circuit breaker is in-memory only and resets on restart (which is
|
||||
fine — startup recovery rescans the database anyway).
|
||||
|
||||
**States:**
|
||||
|
||||
| State | Behavior |
|
||||
| ----------- | -------- |
|
||||
| **Closed** | Normal operation. Deliveries flow through. Consecutive failures are counted. |
|
||||
| **Open** | Target appears down. Deliveries are skipped and rescheduled for after the cooldown. |
|
||||
| **Half-Open** | Cooldown expired. One probe delivery is allowed to test if the target has recovered. |
|
||||
|
||||
**Transitions:**
|
||||
|
||||
```
|
||||
success ┌──────────┐
|
||||
┌────────────────────► │ Closed │ ◄─── probe succeeds
|
||||
│ │ (normal) │
|
||||
│ └────┬─────┘
|
||||
│ │ N consecutive failures
|
||||
│ ▼
|
||||
│ ┌──────────┐
|
||||
│ │ Open │ ◄─── probe fails
|
||||
│ │(tripped) │
|
||||
│ └────┬─────┘
|
||||
│ │ cooldown expires
|
||||
│ ▼
|
||||
│ ┌──────────┐
|
||||
└──────────────────────│Half-Open │
|
||||
│ (probe) │
|
||||
└──────────┘
|
||||
```
|
||||
|
||||
**Defaults:**
|
||||
|
||||
- **Failure threshold:** 5 consecutive failures before opening
|
||||
- **Cooldown:** 30 seconds in open state before probing
|
||||
|
||||
**Scope:** Circuit breakers only apply to **retry** target types. HTTP
|
||||
targets (fire-and-forget), database targets (local operations), and log
|
||||
targets (stdout) do not use circuit breakers.
|
||||
|
||||
When a circuit is open and a new delivery arrives, the engine marks the
|
||||
delivery as `retrying` and schedules a retry timer for after the
|
||||
remaining cooldown period. This ensures no deliveries are lost — they're
|
||||
just delayed until the target is healthy again.
|
||||
|
||||
### Rate Limiting
|
||||
|
||||
Global rate limiting middleware (e.g., per-IP throttling applied at the
|
||||
@@ -606,7 +681,8 @@ webhooker/
|
||||
│ ├── globals/
|
||||
│ │ └── globals.go # Build-time variables (appname, version, arch)
|
||||
│ ├── delivery/
|
||||
│ │ └── engine.go # Event-driven delivery engine (channel + timer based)
|
||||
│ │ ├── engine.go # Event-driven delivery engine (channel + timer based)
|
||||
│ │ └── circuit_breaker.go # Per-target circuit breaker for retry targets
|
||||
│ ├── handlers/
|
||||
│ │ ├── handlers.go # Base handler struct, JSON helpers, template rendering
|
||||
│ │ ├── auth.go # Login, logout handlers
|
||||
@@ -764,6 +840,11 @@ linted, tested, and compiled.
|
||||
Large bodies (≥16KB) are fetched from the per-webhook DB on demand.
|
||||
- [x] Database target type marks delivery as immediately successful
|
||||
(events are already in the per-webhook DB)
|
||||
- [x] Parallel fan-out: all targets for an event are delivered
|
||||
simultaneously in separate goroutines
|
||||
- [x] Circuit breaker for retry targets: tracks consecutive failures
|
||||
per target, opens after 5 failures (30s cooldown), half-open
|
||||
probe to test recovery
|
||||
|
||||
### Remaining: Core Features
|
||||
- [ ] Per-webhook rate limiting in the receiver handler
|
||||
|
||||
Reference in New Issue
Block a user