feat: parallel fan-out delivery + circuit breaker for retry targets
All checks were successful
check / check (push) Successful in 1m52s

- Fan out all targets for an event in parallel goroutines (fire-and-forget)
- Add per-target circuit breaker for retry targets (closed/open/half-open)
- Circuit breaker trips after 5 consecutive failures, 30s cooldown
- Open circuit skips delivery and reschedules after cooldown
- Half-open allows one probe delivery to test recovery
- HTTP/database/log targets unaffected (no circuit breaker)
- Recovery path also fans out in parallel
- Update README with parallel delivery and circuit breaker docs
This commit is contained in:
clawbot
2026-03-01 22:20:33 -08:00
parent 32bd40b313
commit 9b4ae41c44
3 changed files with 407 additions and 92 deletions

View File

@@ -498,14 +498,89 @@ External Service
│ Engine │ (backoff)
└──────┬───────┘
┌────────────────────┼────────────────────┐
▼ ▼
┌────────────┐ ┌────────────┐ ┌────────────┐
│ HTTP Target│ │Retry Target│ │ Log Target │
│ (1 attempt)│ │ (backoff) │ │ (stdout) │
└────────────┘ └────────────┘ └────────────┘
┌─── parallel goroutines (fan-out) ───┐
▼ ▼
┌────────────┐ ┌────────────┐ ┌────────────┐
│ HTTP Target│ │Retry Target│ │ Log Target │
│ (1 attempt)│ │ (backoff + │ │ (stdout) │
└────────────┘ │ circuit │ └────────────┘
│ breaker) │
└────────────┘
```
### Parallel Fan-Out Delivery
When the delivery engine receives a batch of tasks for an event, it
fans out **all targets in parallel** — each `DeliveryTask` is dispatched
in its own goroutine immediately. An HTTP target, a retry target, and
a log target for the same event all start delivering simultaneously
with no sequential bottleneck.
This means:
- **No head-of-line blocking** — a slow HTTP target doesn't delay the
log target or other targets.
- **Maximum throughput** — all targets receive the event as quickly as
possible.
- **Independent results** — each goroutine records its own delivery
result in the per-webhook database without coordination.
- **Fire-and-forget** — the engine doesn't wait for all goroutines to
finish; each delivery is completely independent.
The same parallel fan-out applies to crash recovery: when the engine
restarts and finds pending deliveries in per-webhook databases, it
recovers them and fans them out in parallel just like fresh deliveries.
### Circuit Breaker (Retry Targets)
Retry targets are protected by a **per-target circuit breaker** that
prevents hammering a down target with repeated failed delivery attempts.
The circuit breaker is in-memory only and resets on restart (which is
fine — startup recovery rescans the database anyway).
**States:**
| State | Behavior |
| ----------- | -------- |
| **Closed** | Normal operation. Deliveries flow through. Consecutive failures are counted. |
| **Open** | Target appears down. Deliveries are skipped and rescheduled for after the cooldown. |
| **Half-Open** | Cooldown expired. One probe delivery is allowed to test if the target has recovered. |
**Transitions:**
```
success ┌──────────┐
┌────────────────────► │ Closed │ ◄─── probe succeeds
│ │ (normal) │
│ └────┬─────┘
│ │ N consecutive failures
│ ▼
│ ┌──────────┐
│ │ Open │ ◄─── probe fails
│ │(tripped) │
│ └────┬─────┘
│ │ cooldown expires
│ ▼
│ ┌──────────┐
└──────────────────────│Half-Open │
│ (probe) │
└──────────┘
```
**Defaults:**
- **Failure threshold:** 5 consecutive failures before opening
- **Cooldown:** 30 seconds in open state before probing
**Scope:** Circuit breakers only apply to **retry** target types. HTTP
targets (fire-and-forget), database targets (local operations), and log
targets (stdout) do not use circuit breakers.
When a circuit is open and a new delivery arrives, the engine marks the
delivery as `retrying` and schedules a retry timer for after the
remaining cooldown period. This ensures no deliveries are lost — they're
just delayed until the target is healthy again.
### Rate Limiting
Global rate limiting middleware (e.g., per-IP throttling applied at the
@@ -606,7 +681,8 @@ webhooker/
│ ├── globals/
│ │ └── globals.go # Build-time variables (appname, version, arch)
│ ├── delivery/
│ │ ── engine.go # Event-driven delivery engine (channel + timer based)
│ │ ── engine.go # Event-driven delivery engine (channel + timer based)
│ │ └── circuit_breaker.go # Per-target circuit breaker for retry targets
│ ├── handlers/
│ │ ├── handlers.go # Base handler struct, JSON helpers, template rendering
│ │ ├── auth.go # Login, logout handlers
@@ -764,6 +840,11 @@ linted, tested, and compiled.
Large bodies (≥16KB) are fetched from the per-webhook DB on demand.
- [x] Database target type marks delivery as immediately successful
(events are already in the per-webhook DB)
- [x] Parallel fan-out: all targets for an event are delivered
simultaneously in separate goroutines
- [x] Circuit breaker for retry targets: tracks consecutive failures
per target, opens after 5 failures (30s cooldown), half-open
probe to test recovery
### Remaining: Core Features
- [ ] Per-webhook rate limiting in the receiver handler