clawbot
643077021d
check / check (push) Waiting to run
A target whose circuit breaker had tripped still showed as Active, and its deliveries sat at a bare "retrying" with no attempts. Its row on the webhook page now says deliveries are paused after repeated failures and until when the cooldown ends, adding that one waiting delivery is then sent to test the target; while half-open it says deliveries are held while one tests it, with no time. Waiting deliveries show "next try no earlier than" the later of the cooldown and their own backoff, with the date when not today. The engine gains one read of a breaker's state and remaining cooldown under one lock, and shares the backoff formula. Model: opus-5-5
118 lines
3.5 KiB
Go
118 lines
3.5 KiB
Go
package delivery
|
|
|
|
import (
|
|
"context"
|
|
"net/http"
|
|
"time"
|
|
|
|
"gorm.io/gorm"
|
|
"sneak.berlin/go/webhooker/internal/database"
|
|
)
|
|
|
|
// Scheduler re-enqueues a task for a future delivery attempt.
|
|
// The engine provides one to each target so a target can own
|
|
// its retries durably: it records the attempt, marks the
|
|
// delivery retrying, and asks the Scheduler to deliver the
|
|
// next attempt after delay — exactly what the engine does for
|
|
// its own restart recovery.
|
|
type Scheduler interface {
|
|
ScheduleRetry(task Task, delay time.Duration)
|
|
}
|
|
|
|
// Target delivers an event to one target type. Each type is
|
|
// an implementation. A Target owns its whole delivery: it
|
|
// makes the attempt, records the DeliveryResult and updates
|
|
// the DeliveryStatus, and — for targets that retry — decides
|
|
// whether to retry, computes its own backoff, gates with its
|
|
// own circuit breaker, and reschedules via the injected
|
|
// Scheduler. Fire-and-forget targets simply record a single
|
|
// attempt.
|
|
//
|
|
// An implementation reports each attempt it actually dispatches to
|
|
// Engine.observeAttempt, alongside the DeliveryResult it records for
|
|
// it. Deliver is also entered for attempts that never happen — an
|
|
// open circuit breaker refuses one — so the count cannot be taken
|
|
// from around this call.
|
|
type Target interface {
|
|
Deliver(
|
|
ctx context.Context,
|
|
webhookDB *gorm.DB,
|
|
d *database.Delivery,
|
|
task *Task,
|
|
sched Scheduler,
|
|
)
|
|
}
|
|
|
|
// rescheduler is implemented by targets that own durable
|
|
// retries. The engine's restart recovery and periodic sweep
|
|
// use it to let the target recompute the schedule for an
|
|
// orphaned retrying delivery, keeping the retry schedule
|
|
// target-owned. Fire-and-forget targets do not implement it
|
|
// and their (never-occurring) retrying deliveries are
|
|
// skipped.
|
|
type rescheduler interface {
|
|
// remainingBackoff returns how long to wait before the
|
|
// next attempt of a recovered retrying delivery.
|
|
remainingBackoff(
|
|
webhookDB *gorm.DB,
|
|
deliveryID string,
|
|
attemptNum int,
|
|
) time.Duration
|
|
|
|
// backoffElapsed reports whether the backoff window for
|
|
// the last attempt has already passed, so the periodic
|
|
// sweep can re-enqueue the delivery now.
|
|
backoffElapsed(
|
|
webhookDB *gorm.DB,
|
|
deliveryID string,
|
|
attemptNum int,
|
|
) bool
|
|
}
|
|
|
|
// attemptResult is the outcome of a single delivery attempt,
|
|
// as reported by a target's per-attempt function to the
|
|
// shared retry core.
|
|
type attemptResult struct {
|
|
statusCode int
|
|
respBody string
|
|
duration int64
|
|
success bool
|
|
errMsg string
|
|
}
|
|
|
|
// elapsed returns how long the attempt took. The field is stored in
|
|
// milliseconds because that is what DeliveryResult persists.
|
|
func (r attemptResult) elapsed() time.Duration {
|
|
return time.Duration(r.duration) * time.Millisecond
|
|
}
|
|
|
|
// initTargets builds the target registry, wiring each target
|
|
// to the engine's persistence helpers and giving the HTTP and
|
|
// Slack targets the shared SSRF-safe client. It is called by
|
|
// both New and the test constructors so the registry is
|
|
// always populated.
|
|
func (e *Engine) initTargets(client *http.Client) {
|
|
httpT := &httpTarget{
|
|
httpCore: &httpCore{eng: e},
|
|
client: client,
|
|
}
|
|
|
|
slackT := &slackTarget{
|
|
httpCore: &httpCore{eng: e},
|
|
client: client,
|
|
}
|
|
|
|
dbT := &databaseTarget{eng: e}
|
|
|
|
e.httpTarget = httpT
|
|
e.slackTarget = slackT
|
|
e.dbTarget = dbT
|
|
|
|
e.targets = map[database.TargetType]Target{
|
|
database.TargetTypeHTTP: httpT,
|
|
database.TargetTypeSlack: slackT,
|
|
database.TargetTypeDatabase: dbT,
|
|
database.TargetTypeLog: &logTarget{eng: e},
|
|
}
|
|
}
|