a6a306d810
All checks were successful
check / check (push) Successful in 2m47s
The per-webhook archiveWriter registry in the database delivery target was never evicted, so a deleted webhook's writer -- and any archive file handle open within its debounce window -- lingered for the process lifetime. Separately, expiry pruning ran only when an archive was (re)opened, and reopens only happen on writes, so an archive belonging to a webhook that stopped receiving events kept its expired rows forever. Eviction: a new one-method delivery.WebhookEvictor interface (kept separate from Notifier: archiving lifecycle is not notification) is implemented by the Engine and injected into the handlers. Deleting a webhook, or deleting its last database target, drops the writer from the registry and closes its handle under the writer's own mutex, so eviction can never race an in-flight write. An evicted writer refuses further writes rather than reopening a file nothing holds. The archive file is deliberately left on disk: it is long-term storage an operator may want to keep or move away, and destroying it as a side effect of deleting a webhook would be unrecoverable. Idle sweep: a new ArchiveSweeper, modelled on the event RetentionReaper (fx lifecycle hooks, cancellable context, WaitGroup, ticker loop), prunes archives whose database target declares a positive expiry. It reuses the existing RETENTION_SWEEP_INTERVAL rather than adding a config key. It never creates an archive -- a missing file is skipped, and the reopen uses SQLite mode=rw so the file cannot be conjured even if it disappears mid-sweep -- routes the prune through the per-webhook writer so its mutex orders the sweep against concurrent writes, and leaves the archive closed so the move-the-file-away workflow keeps working. A failure for one webhook is logged and the sweep continues. Archives with no expiry or the expiry "never" are untouched. The sweep loop's context is rooted at context.Background(), not at the fx OnStart hook context. The hook context carries fx's 15 second start timeout, so a loop derived from it is cancelled three quarters of an hour before the first tick under the default one-hour interval, giving a sweeper that never sweeps. OnStop still cancels the loop and waits on the WaitGroup, so shutdown is unchanged. The sweep also never leaves a registry entry behind. Reaching the writer through the ordinary create-and-cache accessor would let a sweep that raced a webhook deletion re-insert a writer for a webhook that no longer exists, which nothing would ever evict again -- the very leak this change closes. An entry the sweep has to create is marked sweep-owned and released when the prune finishes, unless a delivery claimed it meanwhile, in which case it belongs to the registry and an eviction can still reach it. A writer evicted underneath a sweep is an ordinary interleaving and is logged at debug, not error. Each of those guards is pinned by a test that fails when the guard is removed. The adopt-during-sweep window -- a delivery claiming the sweep's own registry entry while that sweep is still running -- is driven directly against the registry, because a delivery placed between two sweeps never reaches the release path at all. The requirement that an idle archive ends the sweep closed is asserted both on a writer proven to hold an open handle beforehand and, end to end, on a delivery-owned entry the sweep keeps, rather than on an entry the sweep has already released and which therefore reports "not open" either way. The "never" expiry short circuit is checked against an archive file that has never been migrated, so any open of it would be observable as a created table.
105 lines
2.9 KiB
Go
105 lines
2.9 KiB
Go
package delivery
|
|
|
|
import (
|
|
"context"
|
|
"net/http"
|
|
"time"
|
|
|
|
"gorm.io/gorm"
|
|
"sneak.berlin/go/webhooker/internal/database"
|
|
)
|
|
|
|
// Scheduler re-enqueues a task for a future delivery attempt.
|
|
// The engine provides one to each target so a target can own
|
|
// its retries durably: it records the attempt, marks the
|
|
// delivery retrying, and asks the Scheduler to deliver the
|
|
// next attempt after delay — exactly what the engine does for
|
|
// its own restart recovery.
|
|
type Scheduler interface {
|
|
ScheduleRetry(task Task, delay time.Duration)
|
|
}
|
|
|
|
// Target delivers an event to one target type. Each type is
|
|
// an implementation. A Target owns its whole delivery: it
|
|
// makes the attempt, records the DeliveryResult and updates
|
|
// the DeliveryStatus, and — for targets that retry — decides
|
|
// whether to retry, computes its own backoff, gates with its
|
|
// own circuit breaker, and reschedules via the injected
|
|
// Scheduler. Fire-and-forget targets simply record a single
|
|
// attempt.
|
|
type Target interface {
|
|
Deliver(
|
|
ctx context.Context,
|
|
webhookDB *gorm.DB,
|
|
d *database.Delivery,
|
|
task *Task,
|
|
sched Scheduler,
|
|
)
|
|
}
|
|
|
|
// rescheduler is implemented by targets that own durable
|
|
// retries. The engine's restart recovery and periodic sweep
|
|
// use it to let the target recompute the schedule for an
|
|
// orphaned retrying delivery, keeping the retry schedule
|
|
// target-owned. Fire-and-forget targets do not implement it
|
|
// and their (never-occurring) retrying deliveries are
|
|
// skipped.
|
|
type rescheduler interface {
|
|
// remainingBackoff returns how long to wait before the
|
|
// next attempt of a recovered retrying delivery.
|
|
remainingBackoff(
|
|
webhookDB *gorm.DB,
|
|
deliveryID string,
|
|
attemptNum int,
|
|
) time.Duration
|
|
|
|
// backoffElapsed reports whether the backoff window for
|
|
// the last attempt has already passed, so the periodic
|
|
// sweep can re-enqueue the delivery now.
|
|
backoffElapsed(
|
|
webhookDB *gorm.DB,
|
|
deliveryID string,
|
|
attemptNum int,
|
|
) bool
|
|
}
|
|
|
|
// attemptResult is the outcome of a single delivery attempt,
|
|
// as reported by a target's per-attempt function to the
|
|
// shared retry core.
|
|
type attemptResult struct {
|
|
statusCode int
|
|
respBody string
|
|
duration int64
|
|
success bool
|
|
errMsg string
|
|
}
|
|
|
|
// initTargets builds the target registry, wiring each target
|
|
// to the engine's persistence helpers and giving the HTTP and
|
|
// Slack targets the shared SSRF-safe client. It is called by
|
|
// both New and the test constructors so the registry is
|
|
// always populated.
|
|
func (e *Engine) initTargets(client *http.Client) {
|
|
httpT := &httpTarget{
|
|
httpCore: &httpCore{eng: e},
|
|
client: client,
|
|
}
|
|
|
|
slackT := &slackTarget{
|
|
httpCore: &httpCore{eng: e},
|
|
client: client,
|
|
}
|
|
|
|
dbT := &databaseTarget{eng: e}
|
|
|
|
e.httpTarget = httpT
|
|
e.dbTarget = dbT
|
|
|
|
e.targets = map[database.TargetType]Target{
|
|
database.TargetTypeHTTP: httpT,
|
|
database.TargetTypeSlack: slackT,
|
|
database.TargetTypeDatabase: dbT,
|
|
database.TargetTypeLog: &logTarget{eng: e},
|
|
}
|
|
}
|