sneak
dfd559417e
All checks were successful
check / check (push) Successful in 3m10s
/metrics carried only the inbound HTTP surface, so a destination failing for an hour, a growing retry backlog and a stuck-open circuit breaker were all invisible: the receive side stays healthy in each case because it is. New internal/metrics registers, on the existing default registry that the go-http-metrics recorder and the promhttp handler already share: - webhooker_events_received_total - webhooker_delivery_attempts_total - webhooker_deliveries_succeeded_total - webhooker_deliveries_failed_total - webhooker_delivery_retries_total - webhooker_delivery_duration_seconds - webhooker_deliveries_pending / _retrying - webhooker_circuit_breakers_open The route mounting is untouched. Every delivery metric carries one label, target_type, whose domain is the four target-type constants; anything outside it collapses to "unknown" so no series can be minted from a UUID. Target ids, event ids and entrypoint ids are deliberately not labels. An attempt is counted, and its duration observed, only where one was actually dispatched — the target's own result path, which is also where the DeliveryResult is written. A delivery an open circuit breaker refuses sends nothing and records no result row; counting it would climb the attempts counter with no traffic behind it and pull the duration quantiles down for as long as the breaker stayed open, moving the metric the wrong way during the outage it exists to reveal. The log and database targets now time their own work, so their result rows carry a real duration too. The outcome counters move after the status row is written rather than before, so a transition the database rejected is never reported as an outcome that happened. The queue-depth gauges are counted out of the per-webhook databases by a 30s sampler rather than tracked as deltas, which would need seeding at startup and would drift on any transition that failed to persist. They publish an "unknown" series from registration: deliveries queued against a target that has since been deleted resolve to the empty type and are folded there, because a backlog behind a deleted target is precisely the one nobody is watching. The open-breaker gauge is recounted from the target's breaker registry on every state change. The orphaned-retry terminal path takes the target type as an argument rather than attaching the loaded target to the delivery. That path loads the delivery without its target relation on purpose: a populated Delivery.Target makes GORM's SaveBeforeAssociations upsert the whole target row on the status UPDATE, writing the plaintext target config — the credential, for a slack target — into the per-webhook events database. A test asserts that path leaves the targets table empty.
117 lines
3.4 KiB
Go
117 lines
3.4 KiB
Go
package delivery
|
|
|
|
import (
|
|
"context"
|
|
"net/http"
|
|
"time"
|
|
|
|
"gorm.io/gorm"
|
|
"sneak.berlin/go/webhooker/internal/database"
|
|
)
|
|
|
|
// Scheduler re-enqueues a task for a future delivery attempt.
|
|
// The engine provides one to each target so a target can own
|
|
// its retries durably: it records the attempt, marks the
|
|
// delivery retrying, and asks the Scheduler to deliver the
|
|
// next attempt after delay — exactly what the engine does for
|
|
// its own restart recovery.
|
|
type Scheduler interface {
|
|
ScheduleRetry(task Task, delay time.Duration)
|
|
}
|
|
|
|
// Target delivers an event to one target type. Each type is
|
|
// an implementation. A Target owns its whole delivery: it
|
|
// makes the attempt, records the DeliveryResult and updates
|
|
// the DeliveryStatus, and — for targets that retry — decides
|
|
// whether to retry, computes its own backoff, gates with its
|
|
// own circuit breaker, and reschedules via the injected
|
|
// Scheduler. Fire-and-forget targets simply record a single
|
|
// attempt.
|
|
//
|
|
// An implementation reports each attempt it actually dispatches to
|
|
// Engine.observeAttempt, alongside the DeliveryResult it records for
|
|
// it. Deliver is also entered for attempts that never happen — an
|
|
// open circuit breaker refuses one — so the count cannot be taken
|
|
// from around this call.
|
|
type Target interface {
|
|
Deliver(
|
|
ctx context.Context,
|
|
webhookDB *gorm.DB,
|
|
d *database.Delivery,
|
|
task *Task,
|
|
sched Scheduler,
|
|
)
|
|
}
|
|
|
|
// rescheduler is implemented by targets that own durable
|
|
// retries. The engine's restart recovery and periodic sweep
|
|
// use it to let the target recompute the schedule for an
|
|
// orphaned retrying delivery, keeping the retry schedule
|
|
// target-owned. Fire-and-forget targets do not implement it
|
|
// and their (never-occurring) retrying deliveries are
|
|
// skipped.
|
|
type rescheduler interface {
|
|
// remainingBackoff returns how long to wait before the
|
|
// next attempt of a recovered retrying delivery.
|
|
remainingBackoff(
|
|
webhookDB *gorm.DB,
|
|
deliveryID string,
|
|
attemptNum int,
|
|
) time.Duration
|
|
|
|
// backoffElapsed reports whether the backoff window for
|
|
// the last attempt has already passed, so the periodic
|
|
// sweep can re-enqueue the delivery now.
|
|
backoffElapsed(
|
|
webhookDB *gorm.DB,
|
|
deliveryID string,
|
|
attemptNum int,
|
|
) bool
|
|
}
|
|
|
|
// attemptResult is the outcome of a single delivery attempt,
|
|
// as reported by a target's per-attempt function to the
|
|
// shared retry core.
|
|
type attemptResult struct {
|
|
statusCode int
|
|
respBody string
|
|
duration int64
|
|
success bool
|
|
errMsg string
|
|
}
|
|
|
|
// elapsed returns how long the attempt took. The field is stored in
|
|
// milliseconds because that is what DeliveryResult persists.
|
|
func (r attemptResult) elapsed() time.Duration {
|
|
return time.Duration(r.duration) * time.Millisecond
|
|
}
|
|
|
|
// initTargets builds the target registry, wiring each target
|
|
// to the engine's persistence helpers and giving the HTTP and
|
|
// Slack targets the shared SSRF-safe client. It is called by
|
|
// both New and the test constructors so the registry is
|
|
// always populated.
|
|
func (e *Engine) initTargets(client *http.Client) {
|
|
httpT := &httpTarget{
|
|
httpCore: &httpCore{eng: e},
|
|
client: client,
|
|
}
|
|
|
|
slackT := &slackTarget{
|
|
httpCore: &httpCore{eng: e},
|
|
client: client,
|
|
}
|
|
|
|
dbT := &databaseTarget{eng: e}
|
|
|
|
e.httpTarget = httpT
|
|
e.dbTarget = dbT
|
|
|
|
e.targets = map[database.TargetType]Target{
|
|
database.TargetTypeHTTP: httpT,
|
|
database.TargetTypeSlack: slackT,
|
|
database.TargetTypeDatabase: dbT,
|
|
database.TargetTypeLog: &logTarget{eng: e},
|
|
}
|
|
}
|