Show a target paused by its circuit breaker (closes #385)

While an http or slack target's circuit breaker is open, the target's
row on the webhook page says its deliveries are paused until the
cooldown ends, then one is sent to test the target. Each of its
retrying deliveries shows as waiting in the event log and on the
event's page, with the earliest it can be tried next: the later of the
cooldown's end and the end of its own backoff, dated when not today
in UTC. While the breaker is half-open, the row says deliveries are
held while one delivery tests the target, with no time, and deliveries
keep their plain status.

The engine gains one read, StateAndCooldown(targetID), taking a
breaker's state and remaining cooldown under one lock; the handlers
reach it through a one-method interface wired like Archives.

Model: opus-5-5

internal/handlers/target_list.go

@@ -24,6 +24,84 @@ type TargetRowView struct {
 	// Archive is a database target's archive file, and nil for a target
 	// of any other type.
 	Archive *ArchiveFileView
+
+	// Paused is set while the target's circuit breaker is turning its
+	// deliveries away, and nil otherwise.
+	Paused *PausedView
+}
+
+// PausedView is a target's circuit breaker turning deliveries away.
+// While the breaker is open, Until is a time in UTC, and Relative how
+// long that is from now: on the target's row, when the cooldown ends;
+// on a delivery, the earliest it can be tried next. While it is
+// half-open both are empty: the cooldown has ended, and the target's
+// deliveries are held while one delivery tests whether the target has
+// recovered.
+type PausedView struct {
+	Until    string
+	Relative string
+}
+
+// pausedView reads the target's circuit breaker for its row, and
+// returns nil when the breaker lets the target's deliveries through.
+func (h *Handlers) pausedView(targetID string) *PausedView {
+	state, cooldown := h.breakers.StateAndCooldown(targetID)
+
+	switch {
+	case state == delivery.CircuitHalfOpen:
+		return &PausedView{}
+	case state == delivery.CircuitOpen && cooldown > 0:
+		return newPausedView(time.Now().Add(cooldown))
+	default:
+		return nil
+	}
+}
+
+// deliveryPausedView reads the circuit breaker of a retrying delivery's
+// target. While it is open, it says the earliest the delivery can be
+// tried next: the later of the cooldown's end and the end of the
+// delivery's own backoff after its last attempt. It is only the
+// earliest: when the cooldown ends, one of the target's waiting
+// deliveries is sent to test it while the others wait at least one more
+// cooldown. Otherwise it returns nil, half-open included, since the
+// delivery may then be the one being sent to test the target.
+func (h *Handlers) deliveryPausedView(
+	targetID string, attempts []deliveryResultRow,
+) *PausedView {
+	state, cooldown := h.breakers.StateAndCooldown(targetID)
+	if state != delivery.CircuitOpen || cooldown <= 0 {
+		return nil
+	}
+
+	next := time.Now().Add(cooldown)
+
+	if len(attempts) > 0 {
+		last := attempts[len(attempts)-1]
+
+		backoffEnd := last.CreatedAt.Add(delivery.Backoff(last.AttemptNum))
+		if backoffEnd.After(next) {
+			next = backoffEnd
+		}
+	}
+
+	return newPausedView(next)
+}
+
+// newPausedView is a PausedView of deliveries paused until the given
+// time. A time not on the current UTC day is written with its date, as
+// the event log writes its times.
+func newPausedView(until time.Time) *PausedView {
+	until = until.UTC()
+
+	layout := time.TimeOnly
+	if until.Format(time.DateOnly) != time.Now().UTC().Format(time.DateOnly) {
+		layout = time.DateTime
+	}
+
+	return &PausedView{
+		Until:    until.Format(layout) + " UTC",
+		Relative: humanize.Time(until),
+	}
 }
 
 // TargetDeliveries is how many of a target's deliveries became
@@ -84,6 +162,8 @@ func (h *Handlers) targetRows(
 		if targets[i].Type == database.TargetTypeDatabase {
 			rows[i].Archive = h.archiveFileView(webhook, &targets[i])
 		}
+
+		rows[i].Paused = h.pausedView(targets[i].ID)
 	}
 
 	return rows