Store events and create deliveries in the delivery engine (closes #544)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
The receiver, resubmit and replay handlers now parse the request and make one call. Engine.StoreEvent stores an event with a pending delivery for each active target and the totals in one transaction, then queues them; Engine.CreateReplay does the same for one replay delivery. The handlers take *delivery.Engine, so the Notifier interface and its bridge are gone, and the engine's notify is no longer exported. Archives and CircuitBreakers stay because the handler tests fake them. The tests that read a queued task moved to the engine's tests; the handler tests use an engine that is never started. Model: opus-5-5
This commit is contained in:
@@ -2153,12 +2153,8 @@ External Service
|
||||
1. Look up Entrypoint by UUID — 404 if unknown,
|
||||
410 if inactive
|
||||
2. Read the body under the 1 MB cap
|
||||
3. Capture full request as Event
|
||||
4. Create Delivery records for each active Target
|
||||
5. Build self-contained delivery.Task structs
|
||||
(target settings + event data inline for
|
||||
bodies < 16 KiB)
|
||||
6. Notify Engine via channel (no DB read needed)
|
||||
3. Hand the full request, as an Event, to
|
||||
Engine.StoreEvent
|
||||
│
|
||||
▼
|
||||
┌──────────────┐
|
||||
@@ -2167,6 +2163,13 @@ External Service
|
||||
│ (worker │
|
||||
│ pool) │
|
||||
└──────┬───────┘
|
||||
4. Store the Event, a pending Delivery for each
|
||||
active Target, and the totals, in one
|
||||
transaction
|
||||
5. Build a delivery.Task for each Delivery
|
||||
(target settings + event data, the body
|
||||
inline when it is under 16 KiB)
|
||||
6. Queue the tasks on the delivery channel
|
||||
│
|
||||
┌── bounded worker pool (N workers) ──┐
|
||||
▼ ▼ ▼
|
||||
@@ -2180,6 +2183,12 @@ External Service
|
||||
└────────────┘
|
||||
```
|
||||
|
||||
A **Resubmit** stores its copy of an event through the same `Engine.StoreEvent`,
|
||||
and a **Replay** adds one delivery of a stored event through
|
||||
`Engine.CreateReplay`, which writes the delivery and its target's totals in one
|
||||
transaction and queues it. Those two methods are the only code that creates a
|
||||
delivery; the handlers parse the request and make the one call.
|
||||
|
||||
### Bounded Worker Pool
|
||||
|
||||
The delivery engine uses a **fixed-size worker pool** (default: 10 workers) to
|
||||
@@ -2189,8 +2198,8 @@ preventing goroutine explosions regardless of queue depth.
|
||||
**Architecture:**
|
||||
|
||||
- **Channels as queues:** Two buffered channels serve as bounded queues: a
|
||||
delivery channel (new tasks from the webhook handler) and a retry channel
|
||||
(tasks from backoff timers). Both are buffered to 10,000.
|
||||
delivery channel (the tasks of deliveries the engine has just stored) and a
|
||||
retry channel (tasks from backoff timers). Both are buffered to 10,000.
|
||||
- **Fan-out via channel, not goroutines:** When an event arrives with multiple
|
||||
targets, each `delivery.Task` is sent to the delivery channel. Workers pick
|
||||
them up and process them — no goroutine-per-target.
|
||||
@@ -3054,6 +3063,7 @@ webhooker/
|
||||
│ │ └── logfield.go # Encoded-byte budget for client-supplied log values
|
||||
│ ├── delivery/
|
||||
│ │ ├── engine.go # Event-driven delivery engine (channel + timer based)
|
||||
│ │ ├── store.go # Storing an event with its deliveries, and a replay, then queueing them
|
||||
│ │ ├── circuit_breaker.go # Per-target circuit breaker for http/slack targets with retries
|
||||
│ │ ├── target.go # Target interface, Task, Scheduler
|
||||
│ │ ├── target_http.go # HTTP target (retries, circuit breaker)
|
||||
@@ -3150,23 +3160,26 @@ Components are wired via Uber fx in this order:
|
||||
10. `metrics.NewRegistry` — The registry `/metrics` serves
|
||||
11. `metrics.New` — The delivery collectors, registered on that registry
|
||||
12. `middleware.New` — HTTP middleware
|
||||
13. `delivery.New` — Event-driven delivery engine
|
||||
13. `delivery.New` — Event-driven delivery engine, which also stores each event
|
||||
with its deliveries
|
||||
14. `archive.NewWriters` — The database targets' archive writers, and the lock
|
||||
every rename of their files runs under
|
||||
15. `archive.NewSweeper` — Periodic pruning of idle archives
|
||||
16. `delivery.Engine` → `delivery.Notifier` — interface bridge
|
||||
17. `archive.Writers` → `archive.Archives` — interface bridge so deleting or
|
||||
16. `archive.Writers` → `archive.Archives` — interface bridge so deleting or
|
||||
renaming a webhook or target reaches its archive files
|
||||
17. `delivery.Engine` → `delivery.CircuitBreakers` — interface bridge so the
|
||||
pages can show a target whose deliveries are paused
|
||||
18. `server.New` — HTTP server and router
|
||||
|
||||
The server starts via
|
||||
`fx.Invoke(func(*server.Server, *delivery.Engine, *database.RetentionReaper, *archive.Sweeper) {})`,
|
||||
which triggers the fx lifecycle hooks in dependency order. The
|
||||
`delivery.Notifier` interface allows the webhook handler to send self-contained
|
||||
`delivery.Task` slices to the engine without a direct package dependency. Each
|
||||
task carries all target settings and event data inline (for bodies under 16 KiB,
|
||||
`delivery.MaxInlineBodySize`), so the engine can deliver without reading from
|
||||
any database — it only writes to record results.
|
||||
which triggers the fx lifecycle hooks in dependency order. The handlers take
|
||||
`*delivery.Engine` itself: the receiver and resubmit handlers store an event
|
||||
through `StoreEvent`, and the replay handler adds a delivery through
|
||||
`CreateReplay`. Each task the engine queues carries the target's settings and
|
||||
the event's data, and the body too when it is under 16 KiB
|
||||
(`delivery.MaxInlineBodySize`); a larger body is read from the event row when
|
||||
the attempt runs.
|
||||
|
||||
### Middleware Stack
|
||||
|
||||
|
||||
Reference in New Issue
Block a user