Add per-delivery replay to the event log (closes #203)

A delivery that exhausted max_retries was failed forever. The event
body is durably stored, so the only way to get it delivered was to
download it and re-POST by hand.

The event log now offers a Replay action on any finished delivery.
Replay creates a NEW pending delivery for the same event and target
and hands it to the delivery engine through the same Notifier the
receiver uses, so it is retried, SSRF-guarded and circuit-broken
exactly as a first attempt. The original delivery's status,
timestamps and recorded attempts are never touched, and what is
re-sent is the stored event body, not the response the original
attempt received.

The target is read as it stands now, including soft-deleted rows so
that a deleted target refuses the replay with a message on the page
instead of erroring or delivering from stale configuration. A
deactivated target and a target id that names nothing refuse the same
way, as does a replay of a delivery the engine has not finished.

Two bounds on replay storms: the route carries a per-client POST rate
limit of 30 per minute, and the handler refuses a replay while an
earlier one for the same event and target is still pending or
retrying.

One new metric, webhooker_delivery_replays_total, on the existing
target_type label. A replay is a real delivery and moves the attempt,
outcome and duration series like any other; this counter is what
separates it from ordinary traffic without adding a dimension to
every existing series.

The delivery row is written with associations omitted and with
neither Event nor Target populated, so no target row reaches the
per-webhook event database.

internal/metrics/metrics.go

@@ -82,6 +82,7 @@ type Set struct {
 	deliveriesSucceeded *prometheus.CounterVec
 	deliveriesFailed    *prometheus.CounterVec
 	deliveryRetries     *prometheus.CounterVec
+	deliveryReplays     *prometheus.CounterVec
 	deliveryDuration    *prometheus.HistogramVec
 	deliveriesPending   *prometheus.GaugeVec
 	deliveriesRetrying  *prometheus.GaugeVec
@@ -149,6 +150,22 @@ func (s *Set) ObserveDeliveryDuration(
 		Observe(d.Seconds())
 }
 
+// DeliveryReplayed counts one delivery an operator replayed from the
+// event log.
+//
+// A replay runs the ordinary engine path, so it already moves the
+// attempt, outcome and duration series exactly as a first delivery
+// does — deliberately, since a replay is a real delivery and hiding it
+// from those would misreport the pipeline. This counter is the one
+// place the two are distinguishable, and it carries the existing
+// target-type label rather than adding a replay dimension to every
+// other series.
+func (s *Set) DeliveryReplayed(t database.TargetType) {
+	s.deliveryReplays.
+		WithLabelValues(normalizeTargetType(t)).
+		Inc()
+}
+
 // DeliveryStatusChanged counts a delivery's transition into a new
 // status. The mapping from status to counter lives here, next to the
 // collectors, so the engine has a single call for every transition it
@@ -271,6 +288,16 @@ func (s *Set) registerCounters(factory promauto.Factory) {
 		},
 		[]string{targetTypeLabel},
 	)
+
+	s.deliveryReplays = factory.NewCounterVec(
+		prometheus.CounterOpts{
+			Namespace: namespace,
+			Name:      "delivery_replays_total",
+			Help: "Deliveries an operator replayed from the " +
+				"event log, by target type.",
+		},
+		[]string{targetTypeLabel},
+	)
 }
 
 func (s *Set) registerGauges(factory promauto.Factory) {
@@ -322,6 +349,7 @@ func (s *Set) initSeries() {
 		s.deliveriesSucceeded.WithLabelValues(label)
 		s.deliveriesFailed.WithLabelValues(label)
 		s.deliveryRetries.WithLabelValues(label)
+		s.deliveryReplays.WithLabelValues(label)
 		s.deliveriesPending.WithLabelValues(label)
 		s.deliveriesRetrying.WithLabelValues(label)
 		s.circuitBreakersOpen.WithLabelValues(label)