Event log: an event's ID can be selected without toggling it (closes #348)

An event's row in the event log was a button element, whose text a browser
does not let be selected. The row is now an element with the button role:
focusable, toggled by Enter and Space, and saying whether it is expanded. A
click on its text toggles only after 500 ms, and the second press of a
double- or triple-click cancels that, so the event does not move under the
pointer while its ID is selected. A click on the caret toggles at once. The
browser test now also clicks the caret, selects the ID by dragging, by a
double-click and drag and by a triple-click, with clicks spaced as a
person's are, and uses the keyboard.

Model: opus-5-5

README.md

@@ -1365,12 +1365,13 @@ hides until the form closes, Cancel hides the form and drops what was typed, as
 does leaving the page and going back to it, and Save changes the description;
 of the recent events on the webhook page only the newest starts expanded, each
 expands and collapses, and Open leads to the event's own page; an event in the
-event log expands and collapses, and so do a delivery's attempts inside it; and
-at phone width the menu button opens and closes the mobile menu. It also fails
-if the browser reports a console warning or error, an uncaught exception, or
-anything the policy refused. `make check`
-and the image build lint it but do not run it, and `make test` leaves it out
-(its file is built only with the `browser` build tag). Run it with
+event log expands and collapses when its row's caret or its ID is clicked, and
+from the keyboard, but not when its ID is selected with the mouse, and a
+delivery's attempts inside it expand and collapse; and at phone width the menu
+button opens and closes the mobile menu. It also fails if the browser reports a
+console warning or error, an uncaught exception, or anything the policy refused.
+`make check` and the image build lint it but do not run it, and `make test`
+leaves it out (its file is built only with the `browser` build tag). Run it with
 `make test-browser` after changing `templates/` or `static/js/`: that builds
 `Dockerfile.browser`, which runs the test in a digest-pinned headless browser
 image, so the host needs no browser.
@@ -1699,7 +1700,6 @@ events should be forwarded.
 | `active`         | boolean    | Whether deliveries are enabled (default: true) |
 | `config`         | JSON text  | Type-specific configuration |
 | `max_retries`    | integer    | Total delivery attempts for `http` and `slack` targets, not retries on top of the first: 0 is a single fire-and-forget attempt with no retries and no circuit breaker, and a value of N makes N attempts in all, with exponential backoff and a per-target circuit breaker. Ignored by `database` and `log` targets |
-| `max_queue_size` | integer    | Stored and shown on the target's detail view, but not enforced anywhere yet: nothing in the delivery engine consults it. Queue depth is set by the two fixed 10,000-entry channels |
 
 **Relations:** Belongs to Webhook. Has many Deliveries.
 
@@ -1857,7 +1857,11 @@ deliver where the destination has since been fixed. A target that has
 been deleted or deactivated therefore refuses the replay with a
 message on the event log rather than delivering from stale
 configuration, and a replay is refused while an earlier one for the
-same event and target is still pending or retrying.
+same event and target is still pending or retrying. A delivery whose
+target has been deleted shows no **Replay** action at all: recreating
+the target makes a new one that the old delivery does not name, so
+**Resubmit** is how that event reaches the webhook's currently active
+targets.
 
 **Resubmit.** Replay recovers one delivery; **resubmit** re-injects one
 EVENT. The event log offers a per-event **Resubmit** action that stores
@@ -2367,6 +2371,20 @@ just delayed until the target is healthy again. A delivery already in
 `retrying` keeps that status without another database write each time
 the breaker turns it away.
 
+While a target's breaker is open, the target's row on the webhook page
+says its deliveries are paused until the cooldown ends, in UTC and as a
+time from now. Each of its `retrying` deliveries shows as waiting in the
+event log and on the event's page, with the earliest it can be tried
+next: the later of the cooldown's end and the end of its own backoff
+after its last attempt. It is only the earliest: when the cooldown ends,
+one of the target's waiting deliveries is sent to test it while the
+others wait at least one more cooldown, as the row also says. A time not
+on the current UTC day is shown with its date. While the breaker is
+half-open, the row says instead that deliveries are held while one
+delivery tests whether the target has recovered, with no time, and the
+target's deliveries show their plain status, since any of them may be
+the one being sent.
+
 ### Metrics
 
 `/metrics` serves one Prometheus registry behind basic auth (see

cmd/webhooker/main.go

@@ -212,6 +212,10 @@ func newApp() *fx.App {
 			// or renaming a webhook or target reaches its archive
 			// files.
 			func(e *delivery.Engine) delivery.Archives { return e },
+			// Wire *delivery.Engine as delivery.CircuitBreakers so
+			// the pages can show a target whose deliveries are
+			// paused.
+			func(e *delivery.Engine) delivery.CircuitBreakers { return e },
 			server.New,
 		),
 		fx.Invoke(

internal/database/model_target.go

@@ -31,8 +31,7 @@ type Target struct {
 
 	// For HTTP targets (max_retries=0 means fire-and-forget,
 	// >0 enables retries with backoff)
-	MaxRetries   int `json:"maxRetries,omitempty"`
-	MaxQueueSize int `json:"maxQueueSize,omitempty"`
+	MaxRetries int `json:"maxRetries,omitempty"`
 
 	// Relations. No model marshals the record it belongs to:
 	// Webhook.Targets leads back here, and the JSON could loop.

internal/delivery/circuit_breaker.go

@@ -102,6 +102,20 @@ func (cb *CircuitBreaker) CooldownRemaining() time.Duration {
 	return remaining
 }
 
+// StateAndCooldown returns the circuit state and, while the circuit is
+// open, what is left of the cooldown, or zero once that has passed.
+// Both are read under one lock, so they always agree.
+func (cb *CircuitBreaker) StateAndCooldown() (CircuitState, time.Duration) {
+	cb.mu.Lock()
+	defer cb.mu.Unlock()
+
+	if cb.state != CircuitOpen {
+		return cb.state, 0
+	}
+
+	return cb.state, max(cb.cooldown-time.Since(cb.lastFailure), 0)
+}
+
 // RecordSuccess records a successful delivery and resets
 // the circuit breaker to closed state.
 func (cb *CircuitBreaker) RecordSuccess() {

internal/delivery/engine.go

@@ -143,6 +143,15 @@ type Archives interface {
 	Rename(targetID, webhookName, targetName string) error
 }
 
+// CircuitBreakers is how the handlers read a target's circuit
+// breaker, so the webhook page and the event log can say that
+// deliveries to the target are paused and until when. Like Archives,
+// it keeps the handlers free of the engine's internals and is
+// trivially faked in tests.
+type CircuitBreakers interface {
+	StateAndCooldown(targetID string) (CircuitState, time.Duration)
+}
+
 // EngineParams are the fx dependencies for the delivery
 // engine.
 type EngineParams struct {
@@ -186,9 +195,11 @@ type Engine struct {
 	// targets maps each target type to its implementation.
 	targets map[database.TargetType]Target
 
-	// httpTarget is retained so tests can reach the HTTP
-	// target's shared client and circuit breakers.
-	httpTarget *httpTarget
+	// httpTarget and slackTarget are retained so StateAndCooldown
+	// can read their circuit breakers, and so tests can reach the
+	// HTTP target's shared client.
+	httpTarget  *httpTarget
+	slackTarget *slackTarget
 
 	// dbTarget is retained so the engine can reach the archive
 	// writer registry for eviction, renames and the idle sweep.
@@ -300,6 +311,28 @@ func (e *Engine) Rename(
 	return e.dbTarget.rename(targetID, webhookName, targetName)
 }
 
+// StateAndCooldown implements CircuitBreakers. It returns the state of
+// the target's circuit breaker and, while the breaker is open, what is
+// left of its cooldown; the cooldown is zero once that has passed and
+// in any other state. A target with no breaker reads as closed with no
+// cooldown, and reading never creates one.
+func (e *Engine) StateAndCooldown(
+	targetID string,
+) (CircuitState, time.Duration) {
+	for _, core := range []*httpCore{
+		e.httpTarget.httpCore, e.slackTarget.httpCore,
+	} {
+		val, ok := core.circuitBreakers.Load(targetID)
+		if ok {
+			cb, _ := val.(*CircuitBreaker)
+
+			return cb.StateAndCooldown()
+		}
+	}
+
+	return CircuitClosed, 0
+}
+
 // ScheduleRetry schedules a task to be re-enqueued onto the
 // retry channel after delay. It implements the Scheduler
 // interface the targets use to own their durable retries.

internal/delivery/engine_test.go

@@ -1018,6 +1018,62 @@ func TestGetCircuitBreaker_CreatesOnDemand(t *testing.T) {
 	)
 }
 
+// TestStateAndCooldown_ReadsHTTPAndSlackBreakers proves the engine
+// reads the state of an http or a slack target's circuit breaker, with
+// what is left of its cooldown while it is open, and no cooldown while
+// it is half-open, once it closes, or for a target with no breaker.
+func TestStateAndCooldown_ReadsHTTPAndSlackBreakers(t *testing.T) {
+	t.Parallel()
+
+	e := testEngine(t, 1)
+
+	httpID := uuid.New().String()
+	slackID := uuid.New().String()
+
+	state, cooldown := e.StateAndCooldown(httpID)
+	assert.Equal(t, delivery.CircuitClosed, state, "no breaker")
+	assert.Zero(t, cooldown, "no breaker")
+
+	httpCB := delivery.NewTestCircuitBreaker(1, time.Hour)
+	e.ExportSetCircuitBreaker(httpID, httpCB)
+
+	slackCB := delivery.NewTestCircuitBreaker(1, time.Hour)
+	e.ExportSetSlackCircuitBreaker(slackID, slackCB)
+
+	httpCB.RecordFailure()
+	slackCB.RecordFailure()
+
+	for _, id := range []string{httpID, slackID} {
+		state, cooldown := e.StateAndCooldown(id)
+		assert.Equal(t, delivery.CircuitOpen, state)
+		assert.Greater(t, cooldown, 59*time.Minute)
+		assert.LessOrEqual(t, cooldown, time.Hour)
+	}
+
+	httpCB.RecordSuccess()
+	slackCB.RecordSuccess()
+
+	for _, id := range []string{httpID, slackID} {
+		state, cooldown := e.StateAndCooldown(id)
+		assert.Equal(t, delivery.CircuitClosed, state, "closed")
+		assert.Zero(t, cooldown, "closed")
+	}
+
+	// A breaker with no cooldown goes half-open on the first Allow
+	// after it trips, letting that one delivery through to test the
+	// target.
+	halfOpenID := uuid.New().String()
+	halfOpenCB := delivery.NewTestCircuitBreaker(1, 0)
+	e.ExportSetCircuitBreaker(halfOpenID, halfOpenCB)
+
+	halfOpenCB.RecordFailure()
+	require.True(t, halfOpenCB.Allow())
+
+	state, cooldown = e.StateAndCooldown(halfOpenID)
+	assert.Equal(t, delivery.CircuitHalfOpen, state)
+	assert.Zero(t, cooldown, "half-open")
+}
+
 func TestParseHTTPConfig_Valid(t *testing.T) {
 	t.Parallel()
 

internal/delivery/export_test.go

@@ -212,6 +212,14 @@ func (e *Engine) ExportSetCircuitBreaker(
 	e.httpTarget.circuitBreakers.Store(targetID, cb)
 }
 
+// ExportSetSlackCircuitBreaker is ExportSetCircuitBreaker for the
+// slack target.
+func (e *Engine) ExportSetSlackCircuitBreaker(
+	targetID string, cb *CircuitBreaker,
+) {
+	e.slackTarget.circuitBreakers.Store(targetID, cb)
+}
+
 // ExportParseHTTPConfig exposes parseHTTPConfig.
 func (e *Engine) ExportParseHTTPConfig(
 	configJSON string,

internal/delivery/target.go

@@ -105,6 +105,7 @@ func (e *Engine) initTargets(client *http.Client) {
 	dbT := &databaseTarget{eng: e}
 
 	e.httpTarget = httpT
+	e.slackTarget = slackT
 	e.dbTarget = dbT
 
 	e.targets = map[database.TargetType]Target{

internal/delivery/target_config_view.go

@@ -173,13 +173,6 @@ func httpConfigFields(t *database.Target) []ConfigField {
 
 	fields = append(fields, maxRetriesField(t))
 
-	if t.MaxQueueSize > 0 {
-		fields = append(fields, ConfigField{
-			Label: "Max Queue Size",
-			Value: strconv.Itoa(t.MaxQueueSize),
-		})
-	}
-
 	return fields
 }
 

internal/delivery/target_config_view_test.go

@@ -197,14 +197,12 @@ func TestNewTargetViews_Slack(t *testing.T) {
 }
 
 // TestNewTargetViews_SlackRetries proves a Slack target shows
-// its retry count the same way an HTTP target does, and no
-// queue size even when one is stored: delivery never reads it.
+// its retry count the same way an HTTP target does.
 func TestNewTargetViews_SlackRetries(t *testing.T) {
 	t.Parallel()
 
 	target := slackTarget()
 	target.MaxRetries = 2
-	target.MaxQueueSize = 100
 
 	view := viewFor(t, target)
 
@@ -226,8 +224,7 @@ func TestNewTargetViews_HTTP(t *testing.T) {
 		Config: `{"url":"` + viewExampleHook + `",` +
 			`"timeout":30,` +
 			`"headers":{"Authorization":"Bearer sekrit"}}`,
-		MaxRetries:   5,
-		MaxQueueSize: 100,
+		MaxRetries: 5,
 	})
 
 	fields := fieldMap(view.Config)
@@ -239,7 +236,6 @@ func TestNewTargetViews_HTTP(t *testing.T) {
 			"Timeout":         "30s",
 			"Headers":         "1 configured",
 			viewMaxRetries:    "5",
-			"Max Queue Size":  "100",
 		},
 		fields,
 	)

internal/delivery/target_http.go

@@ -234,7 +234,7 @@ func (c *httpCore) handleRetry(
 		database.DeliveryStatusRetrying,
 	)
 
-	backoff := calcBackoff(attemptNum)
+	backoff := Backoff(attemptNum)
 
 	retryTask := *task
 	retryTask.AttemptNum = attemptNum + 1
@@ -301,7 +301,7 @@ func (c *httpCore) remainingBackoff(
 		return 0
 	}
 
-	backoff := calcBackoff(attemptNum)
+	backoff := Backoff(attemptNum)
 	elapsed := time.Since(lastResult.CreatedAt)
 	remaining := backoff - elapsed
 
@@ -326,12 +326,14 @@ func (c *httpCore) backoffElapsed(
 		return true
 	}
 
-	backoff := calcBackoff(attemptNum)
+	backoff := Backoff(attemptNum)
 
 	return time.Since(lastResult.CreatedAt) >= backoff
 }
 
-func calcBackoff(attemptNum int) time.Duration {
+// Backoff is how long an http or slack target with retries waits after
+// a delivery's failed attempt attemptNum before trying it again.
+func Backoff(attemptNum int) time.Duration {
 	shift := max(attemptNum-1, 0)
 	shift = min(shift, maxBackoffShift)
 

internal/handlers/delivery_replay.go

@@ -21,7 +21,9 @@ const (
 	// replayTargetDeleted reports a target that once existed and has
 	// since been deleted. Deletes are soft and deliveries carry no
 	// foreign key to the target row, so the history survives its
-	// target and this is the ordinary case for an old event.
+	// target and this is the ordinary case for an old event. The
+	// event log shows no Replay button for such a delivery, so only
+	// a page loaded before the delete reaches this.
 	replayTargetDeleted noticeCode = "replay-target-deleted"
 
 	// replayTargetMissing reports a target id that names no row at

internal/handlers/delivery_replay_test.go

@@ -513,7 +513,11 @@ func TestHandleSourceLogs_RendersReplayControlAndBanner(t *testing.T) {
 	)
 
 	assert.Contains(t, refused, "alert-error")
-	assert.Contains(t, refused, "has been deleted")
+	assert.Contains(
+		t, refused,
+		"has been deleted. Use Resubmit to send the event "+
+			"to the webhook",
+	)
 
 	// An outcome code nobody issued renders no banner at all.
 	unknown := renderSourceLogsPageWithQuery(

internal/handlers/delivery_result_view.go

@@ -1,6 +1,8 @@
 package handlers
 
 import (
+	"time"
+
 	"sneak.berlin/go/webhooker/internal/delivery"
 )
 
@@ -24,8 +26,8 @@ const maxRenderedResponseBytes = 4096
 // bytes rather than characters, and they make SQLite do the
 // cut, so an oversized stored response never becomes a Go
 // string at all.
-const deliveryResultColumns = "delivery_id, attempt_num, success, " +
-	"status_code, error, duration, " +
+const deliveryResultColumns = "delivery_id, attempt_num, created_at, " +
+	"success, status_code, error, duration, " +
 	"substr(cast(response_body as blob), 1, ?) AS response_body, " +
 	"length(cast(response_body as blob)) AS response_bytes"
 
@@ -100,6 +102,7 @@ func (v DeliveryResultView) HasStatusCode() bool {
 type deliveryResultRow struct {
 	DeliveryID    string
 	AttemptNum    int
+	CreatedAt     time.Time
 	Success       bool
 	StatusCode    int
 	Error         string

internal/handlers/handlers.go

@@ -57,19 +57,20 @@ var errVerificationBusy = errors.New(
 type HandlersParams struct {
 	fx.In
 
-	Logger       *logger.Logger
-	Globals      *globals.Globals
-	Config       *config.Config
-	Database     *database.Database
-	WebhookDBMgr *database.WebhookDBManager
-	Healthcheck  *healthcheck.Healthcheck
-	Session      *session.Session
-	Middleware   *middleware.Middleware
-	Notifier     delivery.Notifier
-	Archives     delivery.Archives
-	SSRFGuard    *delivery.Guard
-	Metrics      *metrics.Set
-	Registry     *prometheus.Registry
+	Logger          *logger.Logger
+	Globals         *globals.Globals
+	Config          *config.Config
+	Database        *database.Database
+	WebhookDBMgr    *database.WebhookDBManager
+	Healthcheck     *healthcheck.Healthcheck
+	Session         *session.Session
+	Middleware      *middleware.Middleware
+	Notifier        delivery.Notifier
+	Archives        delivery.Archives
+	CircuitBreakers delivery.CircuitBreakers
+	SSRFGuard       *delivery.Guard
+	Metrics         *metrics.Set
+	Registry        *prometheus.Registry
 }
 
 // Handlers provides HTTP handler methods for all application
@@ -84,6 +85,7 @@ type Handlers struct {
 	mw        *middleware.Middleware
 	notifier  delivery.Notifier
 	archives  delivery.Archives
+	breakers  delivery.CircuitBreakers
 	mtr       *metrics.Set
 	templates map[string]*template.Template
 
@@ -148,6 +150,7 @@ func New(
 	s.mw = params.Middleware
 	s.notifier = params.Notifier
 	s.archives = params.Archives
+	s.breakers = params.CircuitBreakers
 	s.mtr = params.Metrics
 	s.ssrf = params.SSRFGuard
 

internal/handlers/handlers_test.go

@@ -9,6 +9,7 @@ import (
 	"net/http/httptest"
 	"sync"
 	"testing"
+	"time"
 
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
@@ -181,6 +182,40 @@ func (r *recordingArchives) Renames() []archiveRename {
 	return out
 }
 
+// testCircuitBreakers is a delivery.CircuitBreakers that reports, for
+// each target, the circuit state and cooldown a test gave it with Set,
+// and a closed breaker for any other target.
+type testCircuitBreakers struct {
+	mu        sync.Mutex
+	states    map[string]delivery.CircuitState
+	cooldowns map[string]time.Duration
+}
+
+// Set makes the target's breaker read as state, with cooldown left.
+func (b *testCircuitBreakers) Set(
+	targetID string, state delivery.CircuitState, cooldown time.Duration,
+) {
+	b.mu.Lock()
+	defer b.mu.Unlock()
+
+	if b.states == nil {
+		b.states = map[string]delivery.CircuitState{}
+		b.cooldowns = map[string]time.Duration{}
+	}
+
+	b.states[targetID] = state
+	b.cooldowns[targetID] = cooldown
+}
+
+func (b *testCircuitBreakers) StateAndCooldown(
+	targetID string,
+) (delivery.CircuitState, time.Duration) {
+	b.mu.Lock()
+	defer b.mu.Unlock()
+
+	return b.states[targetID], b.cooldowns[targetID]
+}
+
 // newTestApp returns an app whose RequireStart fails the test when
 // starting takes longer than fx's default start timeout of 15s. That
 // limit catches a start that hangs, not a busy host: measured with make
@@ -231,6 +266,12 @@ func newTestAppWithConfig(
 			func(r *recordingArchives) delivery.Archives {
 				return r
 			},
+			func() *testCircuitBreakers {
+				return &testCircuitBreakers{}
+			},
+			func(b *testCircuitBreakers) delivery.CircuitBreakers {
+				return b
+			},
 			metrics.NewRegistry,
 			metrics.New,
 			middleware.New,

internal/handlers/notice.go

@@ -66,7 +66,8 @@ func noticeFor(r *http.Request) *notice {
 		},
 		replayTargetDeleted: {
 			Text: "Not replayed: the target this delivery was for " +
-				"has been deleted. Recreate the target, then replay.",
+				"has been deleted. Use Resubmit to send the event " +
+				"to the webhook's currently active targets.",
 			Failed: true,
 		},
 		replayTargetMissing: {

internal/handlers/source_logs_deleted_target_test.go

@@ -94,6 +94,44 @@ func TestHandleSourceLogs_NamesDeletedTarget(t *testing.T) {
 	)
 }
 
+// TestHandleSourceLogs_OffersNoReplayForDeletedTarget proves a
+// finished delivery offers Replay while its target lives and not
+// once the target is deleted. A replay to a deleted target is always
+// refused, and recreating the target makes a new one that the old
+// delivery does not name.
+func TestHandleSourceLogs_OffersNoReplayForDeletedTarget(t *testing.T) {
+	t.Parallel()
+
+	var (
+		h     *handlers.Handlers
+		sess  *session.Session
+		db    *database.Database
+		dbMgr *database.WebhookDBManager
+	)
+
+	app := newTestApp(t, &h, &sess, &db, &dbMgr)
+	app.RequireStart()
+
+	t.Cleanup(app.RequireStop)
+
+	wh := seedWebhook(t, db)
+	tgt := seedTarget(t, db, wh.ID, database.TargetTypeLog)
+
+	_, failed := seedFailedDelivery(t, dbMgr, wh.ID, tgt.ID)
+	replayForm := `action="/hook/` + wh.ID + `/deliveries/` +
+		failed.ID + `/replay"`
+
+	before := renderSourceLogsPage(t, h, sess, wh.ID)
+	assert.Contains(t, before, replayForm)
+
+	deleteTargetThroughHandler(t, h, sess, wh.ID, tgt.ID)
+
+	after := renderSourceLogsPage(t, h, sess, wh.ID)
+	assert.NotContains(t, after, replayForm)
+	assert.NotContains(t, after, ">Replay<")
+	assert.Contains(t, after, tgt.Name+deletedMarker)
+}
+
 // TestHandleSourceLogs_MasksDeletedTargetConfig proves that
 // naming a deleted target does not widen what the page shows of
 // it: its stored configuration stays masked by exactly the rules

internal/handlers/source_management.go

@@ -109,6 +109,10 @@ type DeliveryView struct {
 	// the middle of Results. The page must show it, or the
 	// bound would hide history rather than fold it.
 	AttemptsOmitted int
+
+	// Paused is set while the delivery is retrying and its
+	// target's circuit breaker is open, and nil otherwise.
+	Paused *PausedView
 }
 
 // eventLogTarget is what the event log needs to know about
@@ -1290,7 +1294,7 @@ func (h *Handlers) eventLogViews(
 	}
 
 	for i := range rows {
-		result[i].Deliveries = newDeliveryViews(
+		result[i].Deliveries = h.newDeliveryViews(
 			eventDeliveries[i], targetMap, attempts,
 		)
 		result[i].ResubmitCount = resubmits[rows[i].ID]
@@ -1414,8 +1418,9 @@ func (h *Handlers) loadDeliveryResults(
 
 // newDeliveryViews projects deliveries for rendering,
 // resolving each one's target to its display-safe view and
-// each one's attempts through that target's redactor.
-func newDeliveryViews(
+// each one's attempts through that target's redactor. A
+// retrying delivery also reads its target's circuit breaker.
+func (h *Handlers) newDeliveryViews(
 	deliveries []database.Delivery,
 	targetMap map[string]eventLogTarget,
 	attempts map[string][]deliveryResultRow,
@@ -1438,6 +1443,12 @@ func newDeliveryViews(
 			AttemptCount:    len(rows),
 			AttemptsOmitted: omitted,
 		}
+
+		if deliveries[i].Status == database.DeliveryStatusRetrying {
+			views[i].Paused = h.deliveryPausedView(
+				deliveries[i].TargetID, rows,
+			)
+		}
 	}
 
 	return views

internal/handlers/target_list.go

@@ -24,6 +24,84 @@ type TargetRowView struct {
 	// Archive is a database target's archive file, and nil for a target
 	// of any other type.
 	Archive *ArchiveFileView
+
+	// Paused is set while the target's circuit breaker is turning its
+	// deliveries away, and nil otherwise.
+	Paused *PausedView
+}
+
+// PausedView is a target's circuit breaker turning deliveries away.
+// While the breaker is open, Until is a time in UTC, and Relative how
+// long that is from now: on the target's row, when the cooldown ends;
+// on a delivery, the earliest it can be tried next. While it is
+// half-open both are empty: the cooldown has ended, and the target's
+// deliveries are held while one delivery tests whether the target has
+// recovered.
+type PausedView struct {
+	Until    string
+	Relative string
+}
+
+// pausedView reads the target's circuit breaker for its row, and
+// returns nil when the breaker lets the target's deliveries through.
+func (h *Handlers) pausedView(targetID string) *PausedView {
+	state, cooldown := h.breakers.StateAndCooldown(targetID)
+
+	switch {
+	case state == delivery.CircuitHalfOpen:
+		return &PausedView{}
+	case state == delivery.CircuitOpen && cooldown > 0:
+		return newPausedView(time.Now().Add(cooldown))
+	default:
+		return nil
+	}
+}
+
+// deliveryPausedView reads the circuit breaker of a retrying delivery's
+// target. While it is open, it says the earliest the delivery can be
+// tried next: the later of the cooldown's end and the end of the
+// delivery's own backoff after its last attempt. It is only the
+// earliest: when the cooldown ends, one of the target's waiting
+// deliveries is sent to test it while the others wait at least one more
+// cooldown. Otherwise it returns nil, half-open included, since the
+// delivery may then be the one being sent to test the target.
+func (h *Handlers) deliveryPausedView(
+	targetID string, attempts []deliveryResultRow,
+) *PausedView {
+	state, cooldown := h.breakers.StateAndCooldown(targetID)
+	if state != delivery.CircuitOpen || cooldown <= 0 {
+		return nil
+	}
+
+	next := time.Now().Add(cooldown)
+
+	if len(attempts) > 0 {
+		last := attempts[len(attempts)-1]
+
+		backoffEnd := last.CreatedAt.Add(delivery.Backoff(last.AttemptNum))
+		if backoffEnd.After(next) {
+			next = backoffEnd
+		}
+	}
+
+	return newPausedView(next)
+}
+
+// newPausedView is a PausedView of deliveries paused until the given
+// time. A time not on the current UTC day is written with its date, as
+// the event log writes its times.
+func newPausedView(until time.Time) *PausedView {
+	until = until.UTC()
+
+	layout := time.TimeOnly
+	if until.Format(time.DateOnly) != time.Now().UTC().Format(time.DateOnly) {
+		layout = time.DateTime
+	}
+
+	return &PausedView{
+		Until:    until.Format(layout) + " UTC",
+		Relative: humanize.Time(until),
+	}
 }
 
 // TargetDeliveries is how many of a target's deliveries became
@@ -84,6 +162,8 @@ func (h *Handlers) targetRows(
 		if targets[i].Type == database.TargetTypeDatabase {
 			rows[i].Archive = h.archiveFileView(webhook, &targets[i])
 		}
+
+		rows[i].Paused = h.pausedView(targets[i].ID)
 	}
 
 	return rows

internal/handlers/target_paused_test.go

@@ -0,0 +1,209 @@
+package handlers_test
+
+import (
+	"net/http"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"gorm.io/gorm/clause"
+	"sneak.berlin/go/webhooker/internal/database"
+	"sneak.berlin/go/webhooker/internal/delivery"
+	"sneak.berlin/go/webhooker/internal/handlers"
+	"sneak.berlin/go/webhooker/internal/session"
+)
+
+// cooldownEnds is how the pages write the end of a paused target's
+// breaker's cooldown: the time in UTC, with its date when that falls on
+// another UTC day, then how long that is from now.
+const cooldownEnds = `(\d{4}-\d\d-\d\d )?\d\d:\d\d:\d\d UTC ` +
+	`\(\d+ seconds from now\)`
+
+// TestPausedTarget_ShownUntilBreakerCloses takes an http target's
+// circuit breaker from open through half-open to closed.
+//
+// Open, the target's row on the webhook page says its deliveries are
+// paused and until when, and each retrying delivery says it is waiting
+// and why in the event log and on the event's page, with the earliest
+// it can be tried next: the later of the cooldown's end and the end of
+// its own backoff, with the date when that is another UTC day.
+// Half-open, the row says deliveries are held while one delivery tests
+// the target, with no time, and no delivery says it is waiting. Closed,
+// the pages say neither. The delivered delivery and the log target are
+// shown as before throughout.
+func TestPausedTarget_ShownUntilBreakerCloses(t *testing.T) {
+	t.Parallel()
+
+	var (
+		h        *handlers.Handlers
+		sess     *session.Session
+		db       *database.Database
+		dbMgr    *database.WebhookDBManager
+		breakers *testCircuitBreakers
+	)
+
+	app := newTestApp(t, &h, &sess, &db, &dbMgr, &breakers)
+	app.RequireStart()
+
+	t.Cleanup(app.RequireStop)
+
+	wh := seedWebhook(t, db)
+	target := seedTarget(t, db, wh.ID, database.TargetTypeHTTP)
+	seedTarget(t, db, wh.ID, database.TargetTypeLog)
+
+	retrying := seedStoredEvent(t, dbMgr, wh.ID, `{"n":1}`)
+	addDelivery(t, dbMgr, wh.ID, retrying.ID, target.ID,
+		database.DeliveryStatusRetrying)
+
+	delivered := seedStoredEvent(t, dbMgr, wh.ID, `{"n":2}`)
+	addDelivery(t, dbMgr, wh.ID, delivered.ID, target.ID,
+		database.DeliveryStatusDelivered)
+
+	// This delivery's 18th attempt failed a minute ago, so its own
+	// backoff ends over a day from now: long after the cooldown, and on
+	// another UTC day, so the page shows the date.
+	backedOff := seedStoredEvent(t, dbMgr, wh.ID, `{"n":3}`)
+	backedOffID := addDelivery(t, dbMgr, wh.ID, backedOff.ID, target.ID,
+		database.DeliveryStatusRetrying)
+
+	failedAt := time.Now().Add(-time.Minute).Truncate(time.Second)
+	addFailedAttempt(t, dbMgr, wh.ID, backedOffID, 18, failedAt)
+
+	backoffEnds := failedAt.Add(delivery.Backoff(18)).UTC().
+		Format("2006-01-02 15:04:05") + " UTC (1 day from now)"
+
+	const waiting = "waiting: target paused after repeated failures, " +
+		"next try no earlier than "
+
+	breakers.Set(target.ID, delivery.CircuitOpen, 30*time.Second)
+
+	list := targetList(t, renderSourceDetailPage(t, h, sess, wh.ID))
+	assert.Regexp(t, "t-http http Active Edit Deactivate Delete "+
+		"Deliveries Paused: after repeated failures, until "+cooldownEnds+
+		", then one waiting delivery is sent to test the target while "+
+		"the others wait at least one more cooldown", list)
+	assert.Equal(t, 1, strings.Count(list, "Paused"))
+
+	log := renderSourceLogsPage(t, h, sess, wh.ID)
+	assert.Equal(t, 2, strings.Count(log, "t-http: waiting"))
+	assert.Contains(t, log, "t-http: delivered")
+	assert.Regexp(t, waiting+cooldownEnds, log)
+	assert.Contains(t, log, waiting+backoffEnds)
+	assert.NotContains(t, log, "retrying")
+
+	page := eventPage(t, h, sess, wh.ID, retrying.ID)
+	assert.Regexp(t, waiting+cooldownEnds, page)
+	assert.NotContains(t, page, "retrying")
+
+	page = eventPage(t, h, sess, wh.ID, backedOff.ID)
+	assert.Contains(t, page, waiting+backoffEnds)
+	assert.NotContains(t, page, "seconds from now")
+
+	breakers.Set(target.ID, delivery.CircuitHalfOpen, 0)
+
+	list = targetList(t, renderSourceDetailPage(t, h, sess, wh.ID))
+	assert.Contains(t, list, "t-http http Active Edit Deactivate Delete "+
+		"Deliveries Paused: held while one delivery tests whether the "+
+		"target has recovered")
+	assert.NotContains(t, list, "UTC")
+
+	assertRetryingNotWaiting(t, h, sess, wh.ID, retrying, backedOff)
+
+	breakers.Set(target.ID, delivery.CircuitClosed, 0)
+
+	list = targetList(t, renderSourceDetailPage(t, h, sess, wh.ID))
+	assert.NotContains(t, list, "Paused")
+
+	assertRetryingNotWaiting(t, h, sess, wh.ID, retrying, backedOff)
+}
+
+// assertRetryingNotWaiting checks that the event log and each event's
+// page show the http target's delivery of the event as retrying, and
+// none of them as waiting.
+func assertRetryingNotWaiting(
+	t *testing.T,
+	h *handlers.Handlers,
+	sess *session.Session,
+	webhookID string,
+	events ...*database.Event,
+) {
+	t.Helper()
+
+	log := renderSourceLogsPage(t, h, sess, webhookID)
+	assert.Equal(t, len(events), strings.Count(log, "t-http: retrying"))
+	assert.NotContains(t, log, "waiting")
+
+	for _, event := range events {
+		page := eventPage(t, h, sess, webhookID, event.ID)
+		assert.Contains(t, page, ">retrying</span>")
+		assert.NotContains(t, page, "waiting")
+	}
+}
+
+// addDelivery records a delivery of the event to the target, with the
+// given status, in the webhook's own database, and returns its ID.
+func addDelivery(
+	t *testing.T,
+	dbMgr *database.WebhookDBManager,
+	webhookID, eventID, targetID string,
+	status database.DeliveryStatus,
+) string {
+	t.Helper()
+
+	webhookDB, err := dbMgr.GetDB(webhookID)
+	require.NoError(t, err)
+
+	dlv := &database.Delivery{
+		EventID:  eventID,
+		TargetID: targetID,
+		Status:   status,
+	}
+
+	require.NoError(t, webhookDB.Omit(clause.Associations).Create(
+		dlv,
+	).Error)
+
+	return dlv.ID
+}
+
+// addFailedAttempt records the delivery's failed attempt attemptNum,
+// made at the given time.
+func addFailedAttempt(
+	t *testing.T,
+	dbMgr *database.WebhookDBManager,
+	webhookID, deliveryID string,
+	attemptNum int,
+	at time.Time,
+) {
+	t.Helper()
+
+	webhookDB, err := dbMgr.GetDB(webhookID)
+	require.NoError(t, err)
+
+	require.NoError(t, webhookDB.Omit(clause.Associations).Create(
+		&database.DeliveryResult{
+			BaseModel:  database.BaseModel{CreatedAt: at},
+			DeliveryID: deliveryID,
+			AttemptNum: attemptNum,
+			Error:      "connection refused",
+		},
+	).Error)
+}
+
+// eventPage runs the real event page handler and returns the
+// rendered HTML.
+func eventPage(
+	t *testing.T,
+	h *handlers.Handlers,
+	sess *session.Session,
+	webhookID, eventID string,
+) string {
+	t.Helper()
+
+	w := serveEventPage(t, h, sess, webhookID, eventID)
+	require.Equal(t, http.StatusOK, w.Code)
+
+	return w.Body.String()
+}

internal/resetpw/resetpw_test.go

@@ -11,6 +11,7 @@ import (
 	"path/filepath"
 	"strings"
 	"testing"
+	"time"
 
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
@@ -142,6 +143,14 @@ func (n *noopArchives) Rename(_, _, _ string) error {
 	return nil
 }
 
+type noopCircuitBreakers struct{}
+
+func (n *noopCircuitBreakers) StateAndCooldown(
+	string,
+) (delivery.CircuitState, time.Duration) {
+	return delivery.CircuitClosed, 0
+}
+
 // newServerApp starts the real login path against dir: the handlers,
 // the middleware that bounds password verification, the session store
 // and the database, exactly as internal/handlers builds them.
@@ -174,6 +183,9 @@ func newServerApp(
 			session.New,
 			func() delivery.Notifier { return &noopNotifier{} },
 			func() delivery.Archives { return &noopArchives{} },
+			func() delivery.CircuitBreakers {
+				return &noopCircuitBreakers{}
+			},
 			metrics.NewRegistry,
 			metrics.New,
 			middleware.New,

internal/server/alpine_browser_test.go

@@ -18,10 +18,13 @@ import (
 	"time"
 
 	"github.com/chromedp/cdproto/browser"
+	"github.com/chromedp/cdproto/dom"
+	"github.com/chromedp/cdproto/input"
 	"github.com/chromedp/cdproto/log"
 	"github.com/chromedp/cdproto/network"
 	"github.com/chromedp/cdproto/runtime"
 	"github.com/chromedp/chromedp"
+	"github.com/chromedp/chromedp/kb"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 	"gorm.io/gorm/clause"
@@ -40,6 +43,16 @@ const (
 	phoneWidth  = 390
 	phoneHeight = 844
 
+	// A window short enough that the event log scrolls with its last
+	// event expanded, and tall enough to show all of that event.
+	shortWidth  = 1024
+	shortHeight = 450
+
+	// A person's double- or triple-click: each press is held a tenth of a
+	// second, and the next press comes a quarter second after the release.
+	pressHeld     = 100 * time.Millisecond
+	betweenClicks = 250 * time.Millisecond
+
 	// olderBody is the body of the event received before the newest.
 	olderBody = "the older event"
 )
@@ -59,7 +72,7 @@ func TestAlpineRunsUnderTheSecurityPolicy(t *testing.T) {
 	t.Cleanup(srv.Close)
 
 	userID, _ := env.seedUser(t, "browser", "browser-password")
-	webhook, event, target := seedBrowserWebhook(t, env, userID)
+	webhook, older, event, target := seedBrowserWebhook(t, env, userID)
 
 	require.NoError(t, chromedp.Run(
 		ctx, setCookies(srv.URL, env.authCookies(t, userID, "browser")),
@@ -80,10 +93,10 @@ func TestAlpineRunsUnderTheSecurityPolicy(t *testing.T) {
 	checkCopy(ctx, t, page)
 	checkEntrypointEdit(ctx, t, page, page+"/events")
 	checkRecentEvents(ctx, t, page)
-	checkEventLog(ctx, t, page+"/events", event.ID, target.Name)
 	checkArchiveChoice(ctx, t, srv.URL+"/hooks/new", page)
 	checkNewWebhookTargets(ctx, t, env, srv.URL+"/hooks/new")
 	checkRefusedNewWebhook(ctx, t, srv.URL+"/hooks/new")
+	checkEventLog(ctx, t, page+"/events", event.ID, older.ID, target.Name)
 	checkMobileMenu(ctx, t, page)
 
 	assert.Empty(t, problems(), "the browser reported problems")
@@ -91,11 +104,11 @@ func TestAlpineRunsUnderTheSecurityPolicy(t *testing.T) {
 
 // seedBrowserWebhook seeds the webhook the browser test loads, owned by
 // userID: an entrypoint, two events, and a target whose delivery of the
-// newer event failed once with a 502. It returns the webhook, the newer
-// event and the target.
+// newer event failed once with a 502. It returns the webhook, the older
+// and the newer event, and the target.
 func seedBrowserWebhook(
 	t *testing.T, env *testEnv, userID string,
-) (*database.Webhook, *database.Event, *database.Target) {
+) (*database.Webhook, *database.Event, *database.Event, *database.Target) {
 	t.Helper()
 
 	webhook := env.seedWebhook(t, userID)
@@ -106,7 +119,7 @@ func seedBrowserWebhook(
 			Active:    true,
 		},
 	).Error)
-	env.seedEvent(t, webhook.ID, olderBody)
+	older := env.seedEvent(t, webhook.ID, olderBody)
 	event := env.seedEvent(t, webhook.ID, `{"hello":"browser"}`)
 	target := env.seedTarget(t, webhook.ID)
 	dlv := env.seedFailedDelivery(t, webhook.ID, event.ID, target.ID)
@@ -121,7 +134,7 @@ func seedBrowserWebhook(
 		},
 	).Error)
 
-	return webhook, event, target
+	return webhook, older, event, target
 }
 
 // startBrowser starts a headless browser for one test. It returns the
@@ -752,18 +765,30 @@ func checkRecentEvents(ctx context.Context, t *testing.T, url string) {
 		"the event's own page does not show its body")
 }
 
-// checkEventLog loads the event log and checks that clicking an event's
-// row expands it, that in there clicking its delivery shows the
-// delivery's attempts and clicking again hides them, and that clicking
-// the event's row again collapses it.
+// checkEventLog loads the event log and checks an event's row. Clicking
+// its ID expands the event, and in there clicking its delivery shows the
+// delivery's attempts and clicking again hides them. Clicking the row's
+// caret collapses the event, clicking it again expands it, and clicking
+// the ID again collapses it. While the event is expanded the row says so
+// and its caret is turned up, and while it is collapsed neither. It then
+// runs checkEventSelection on the log's last event, lastEventID, and
+// checkEventKeyboard on eventID.
 func checkEventLog(
-	ctx context.Context, t *testing.T, url, eventID, targetName string,
+	ctx context.Context,
+	t *testing.T,
+	url, eventID, lastEventID, targetName string,
 ) {
 	t.Helper()
 
-	// The event's row shows its ID, and its Resubmit form is in the part
-	// that expands. The delivery's row there shows the target's name.
-	eventRow := `//span[text()="` + eventID + `"]`
+	// The event's row shows its ID and ends with its caret, which turns
+	// up with Tailwind's rotate-180 class, and its Resubmit form is in
+	// the part that expands. The delivery's row there shows the target's
+	// name.
+	id := `//span[text()="` + eventID + `"]`
+	row := id + `/ancestor::div[@role="button"]`
+	caret := row + `//*[local-name()="svg"]`
+	caretUp := caret + `[contains(@class, "rotate-180")]`
+	caretDown := caret + `[not(contains(@class, "rotate-180"))]`
 	expanded := `form[action$="/` + eventID + `/resubmit"]`
 	deliveryRow := `//span[text()="` + targetName + `"]`
 	attempt := `//span[text()="Attempt 1"]`
@@ -772,8 +797,13 @@ func checkEventLog(
 
 	assert.True(t, hidden(ctx, expanded), "the event starts expanded")
 
-	click(ctx, t, eventRow)
-	assert.True(t, shown(ctx, expanded), "clicking the event does not expand it")
+	click(ctx, t, id)
+	assert.True(t, shown(ctx, expanded),
+		"clicking the event's ID does not expand it")
+	assert.True(t, shown(ctx, row+`[@aria-expanded="true"]`),
+		"the expanded event's row does not say it is expanded")
+	assert.True(t, shown(ctx, caretUp),
+		"the expanded event's caret does not turn up")
 
 	assert.True(t, hidden(ctx, attempt), "the delivery's attempts start shown")
 
@@ -785,9 +815,232 @@ func checkEventLog(
 	assert.True(t, hidden(ctx, attempt),
 		"clicking the delivery again does not hide its attempts")
 
-	click(ctx, t, eventRow)
+	click(ctx, t, caret)
 	assert.True(t, hidden(ctx, expanded),
-		"clicking the event again does not collapse it")
+		"clicking the caret does not collapse the event")
+	assert.True(t, shown(ctx, row+`[@aria-expanded="false"]`),
+		"the collapsed event's row does not say it is collapsed")
+	assert.True(t, shown(ctx, caretDown),
+		"the collapsed event's caret stays turned up")
+
+	click(ctx, t, caret)
+	assert.True(t, shown(ctx, expanded),
+		"clicking the caret again does not expand the event")
+
+	click(ctx, t, id)
+	assert.True(t, hidden(ctx, expanded),
+		"clicking the event's ID again does not collapse it")
+
+	checkEventSelection(ctx, t, url, lastEventID)
+	checkEventKeyboard(ctx, t, url, eventID)
+}
+
+// checkEventSelection loads the event log in a short window and checks
+// that selecting the ID of its last event, eventID, with the mouse leaves
+// the event as it was, and that its caret toggles it at once. Dragging
+// over the ID leaves the event collapsed, and the caret's click expands
+// it at once. With the page then scrolled to its end, a double-click on
+// the ID that goes on to drag along it, and a triple-click on it, each
+// leave the event expanded and select that ID. Had a click there
+// collapsed the event, the page would have got shorter and moved under
+// the pointer before the next click.
+func checkEventSelection(
+	ctx context.Context, t *testing.T, url, eventID string,
+) {
+	t.Helper()
+
+	id := `//span[text()="` + eventID + `"]`
+	row := id + `/ancestor::div[@role="button"]`
+	caret := row + `//*[local-name()="svg"]`
+	expanded := `form[action$="/` + eventID + `/resubmit"]`
+
+	var (
+		selected, state string
+		hasState        bool
+		scrolled        float64
+	)
+
+	// What is selected, and whether the event's row says it is expanded.
+	read := chromedp.Tasks{
+		chromedp.Evaluate(`window.getSelection().toString()`, &selected),
+		chromedp.AttributeValue(
+			row, "aria-expanded", &state, &hasState, chromedp.BySearch,
+		),
+	}
+
+	// A click on the ID toggles the event half a second after it, so a
+	// check that selecting the ID did not toggle it waits a second first.
+	settle := chromedp.Sleep(time.Second)
+
+	// The double-click and the triple-click each start with nothing
+	// selected, so that their first click waits to toggle the event.
+	clearSelection := chromedp.Evaluate(
+		`window.getSelection().removeAllRanges()`, nil,
+	)
+
+	require.NoError(t, chromedp.Run(
+		ctx, chromedp.EmulateViewport(shortWidth, shortHeight), loadPage(url),
+	))
+
+	selectText(ctx, t, id)
+	require.NoError(t, chromedp.Run(ctx, settle, read))
+	assert.Equal(t, eventID, selected, "the event's ID cannot be selected")
+	require.True(t, hasState, "the event's row does not say if it is expanded")
+	assert.Equal(t, "false", state, "selecting the event's ID expands it")
+
+	click(ctx, t, caret)
+	require.NoError(t, chromedp.Run(ctx, read))
+	assert.Equal(t, "true", state,
+		"clicking the caret does not expand the event at once")
+
+	// The row says it is expanded before its expanded part is shown, so
+	// the scroll waits for that part, to end at the expanded page's end.
+	require.True(t, shown(ctx, expanded),
+		"clicking the caret does not show the event's expanded part")
+
+	require.NoError(t, chromedp.Run(ctx, chromedp.Evaluate(
+		`window.scrollTo(0, document.body.scrollHeight); window.scrollY`,
+		&scrolled,
+	)))
+	require.Positive(t, scrolled, "the event log does not scroll")
+
+	require.NoError(t, chromedp.Run(ctx, clearSelection))
+	doubleClickAndDrag(ctx, t, id)
+	require.NoError(t, chromedp.Run(ctx, settle, read))
+	assert.Contains(t, selected, eventID,
+		"a double-click and drag does not select the event's ID")
+	assert.Equal(t, "true", state,
+		"a double-click and drag over the event's ID collapses it")
+
+	require.NoError(t, chromedp.Run(ctx, clearSelection))
+	tripleClick(ctx, t, id)
+	require.NoError(t, chromedp.Run(ctx, settle, read))
+	assert.Contains(t, selected, eventID,
+		"a triple-click does not select the event's ID")
+	assert.Equal(t, "true", state,
+		"a triple-click selecting the event's ID collapses it")
+}
+
+// checkEventKeyboard loads the event log and checks that Tab from the
+// page's Back link reaches the event's row, the first after it, and that
+// Enter then expands the event and Space collapses it.
+func checkEventKeyboard(
+	ctx context.Context, t *testing.T, url, eventID string,
+) {
+	t.Helper()
+
+	back := `//a[contains(text(), "Back to")]`
+	expanded := `form[action$="/` + eventID + `/resubmit"]`
+
+	var focused string
+
+	require.NoError(t, chromedp.Run(
+		ctx,
+		loadPage(url),
+		chromedp.Focus(back, chromedp.BySearch),
+		chromedp.KeyEvent(kb.Tab),
+		chromedp.Evaluate(`document.activeElement.textContent`, &focused),
+	))
+	require.Contains(t, focused, eventID,
+		"Tab from the Back link does not reach the event's row")
+
+	require.NoError(t, chromedp.Run(ctx, chromedp.KeyEvent(kb.Enter)))
+	assert.True(t, shown(ctx, expanded), "Enter does not expand the event")
+
+	require.NoError(t, chromedp.Run(ctx, chromedp.KeyEvent(" ")))
+	assert.True(t, hidden(ctx, expanded), "Space does not collapse the event")
+}
+
+// selectText selects the text of the element matching an XPath
+// expression as a person does with the mouse: pressing the button at the
+// text's start, moving to its end and releasing it there.
+func selectText(ctx context.Context, t *testing.T, xpath string) {
+	t.Helper()
+
+	left, right, y := textEnds(ctx, t, xpath)
+
+	require.NoError(t, chromedp.Run(
+		ctx, press(left, y, 1), drag(right, y), release(right, y, 1),
+	))
+}
+
+// doubleClickAndDrag double-clicks the start of the text of the element
+// matching an XPath expression, which selects its first word, and keeps
+// the button down to drag to the text's end, which selects it word by
+// word. It holds the button for a second, longer than a single click on
+// an event's row waits before it toggles the event.
+func doubleClickAndDrag(ctx context.Context, t *testing.T, xpath string) {
+	t.Helper()
+
+	left, right, y := textEnds(ctx, t, xpath)
+
+	require.NoError(t, chromedp.Run(
+		ctx,
+		press(left, y, 1), chromedp.Sleep(pressHeld), release(left, y, 1),
+		chromedp.Sleep(betweenClicks),
+		press(left, y, 2), drag(right, y), chromedp.Sleep(time.Second),
+		release(right, y, 2),
+	))
+}
+
+// tripleClick clicks three times in the middle of the text of the
+// element matching an XPath expression, as a person does to select a
+// whole line of text. The browser selects a word on the second click and
+// the whole paragraph on the third.
+func tripleClick(ctx context.Context, t *testing.T, xpath string) {
+	t.Helper()
+
+	left, right, y := textEnds(ctx, t, xpath)
+	x := (left + right) / 2
+
+	require.NoError(t, chromedp.Run(
+		ctx,
+		press(x, y, 1), chromedp.Sleep(pressHeld), release(x, y, 1),
+		chromedp.Sleep(betweenClicks),
+		press(x, y, 2), chromedp.Sleep(pressHeld), release(x, y, 2),
+		chromedp.Sleep(betweenClicks),
+		press(x, y, 3), chromedp.Sleep(pressHeld), release(x, y, 3),
+	))
+}
+
+// textEnds returns where on screen the text of the element matching an
+// XPath expression starts and ends, just inside its left and right
+// edges, and the height of its middle: in that order, the x of its
+// start, the x of its end, and the y of both.
+func textEnds(
+	ctx context.Context, t *testing.T, xpath string,
+) (float64, float64, float64) {
+	t.Helper()
+
+	var box *dom.BoxModel
+
+	require.NoError(t, chromedp.Run(
+		ctx, chromedp.Dimensions(xpath, &box, chromedp.BySearch),
+	))
+
+	// The content box's corners, clockwise from its top left.
+	return box.Content[0] + 1, box.Content[2] - 1,
+		(box.Content[1] + box.Content[5]) / 2
+}
+
+// press presses the left mouse button at x, y, as the nth click of a
+// double- or triple-click.
+func press(x, y float64, nth int64) *input.DispatchMouseEventParams {
+	return input.DispatchMouseEvent(input.MousePressed, x, y).
+		WithButton(input.Left).WithButtons(1).WithClickCount(nth)
+}
+
+// drag moves the pointer to x, y with the left mouse button down.
+func drag(x, y float64) *input.DispatchMouseEventParams {
+	return input.DispatchMouseEvent(input.MouseMoved, x, y).
+		WithButton(input.Left).WithButtons(1)
+}
+
+// release releases the left mouse button at x, y, as the nth click of a
+// double- or triple-click.
+func release(x, y float64, nth int64) *input.DispatchMouseEventParams {
+	return input.DispatchMouseEvent(input.MouseReleased, x, y).
+		WithButton(input.Left).WithClickCount(nth)
 }
 
 // The parts of the new webhook page the checks below find and click.

internal/server/routes_test.go

@@ -62,6 +62,17 @@ func (e *noopArchives) Rename(_, _, _ string) error {
 	return nil
 }
 
+// noopCircuitBreakers satisfies handlers.New's
+// delivery.CircuitBreakers dependency with no target's deliveries
+// paused.
+type noopCircuitBreakers struct{}
+
+func (b *noopCircuitBreakers) StateAndCooldown(
+	string,
+) (delivery.CircuitState, time.Duration) {
+	return delivery.CircuitClosed, 0
+}
+
 // testEnv is the real router from routes.go plus the collaborators
 // tests need to seed users and forge sessions.
 type testEnv struct {
@@ -126,6 +137,9 @@ func newTestEnvWithConfig(
 			session.New,
 			func() delivery.Notifier { return &noopNotifier{} },
 			func() delivery.Archives { return &noopArchives{} },
+			func() delivery.CircuitBreakers {
+				return &noopCircuitBreakers{}
+			},
 			metrics.NewRegistry,
 			metrics.New,
 			middleware.New,

static/js/app.js

@@ -77,12 +77,36 @@ document.addEventListener("alpine:init", function () {
     window.Alpine.data("collapsible", function () {
         return {
             open: false,
+            // The timer of the toggle a single click is waiting to make.
+            pendingToggle: null,
             init() {
                 this.open = this.$root.hasAttribute("data-open");
             },
             toggle() {
                 this.open = !this.open;
             },
+            // Toggles on a click, except one that selects text, such as
+            // selecting an event's ID to copy it. A single click toggles
+            // only after 500 ms, the usual double-click interval, and the
+            // second press of a double- or triple-click cancels that (see
+            // cancelPendingToggle), so nothing moves under the pointer
+            // while it selects text.
+            toggleUnlessSelecting(event) {
+                if (
+                    event.detail === 1 &&
+                    window.getSelection().toString() === ""
+                ) {
+                    this.pendingToggle = setTimeout(() => this.toggle(), 500);
+                }
+            },
+            // Runs when the mouse button goes down, so the second press
+            // of a double- or triple-click cancels the toggle its first
+            // click is waiting to make, however long that press lasts.
+            cancelPendingToggle(event) {
+                if (event.detail > 1) {
+                    clearTimeout(this.pendingToggle);
+                }
+            },
             get closed() {
                 return !this.open;
             },

templates/event_detail.html

@@ -69,7 +69,7 @@
                 <div class="flex flex-wrap items-center justify-between gap-3">
                     <span class="text-sm text-gray-700">{{.Target.DisplayName}}</span>
                     <span class="flex flex-wrap items-center gap-3">
-                        <span class="text-xs {{if eq .Status "delivered"}}text-green-600{{else if eq .Status "failed"}}text-red-600{{else if eq .Status "retrying"}}text-yellow-600{{else}}text-gray-400{{end}}">{{.Status}}</span>
+                        <span class="text-xs {{if eq .Status "delivered"}}text-green-600{{else if eq .Status "failed"}}text-red-600{{else if eq .Status "retrying"}}text-yellow-600{{else}}text-gray-400{{end}}">{{with .Paused}}waiting: target paused after repeated failures, next try no earlier than {{.Until}} ({{.Relative}}){{else}}{{.Status}}{{end}}</span>
                         <span class="text-xs text-gray-400">{{.AttemptCount}} attempt{{if ne .AttemptCount 1}}s{{end}}</span>
                     </span>
                 </div>

templates/source_detail.html

@@ -255,6 +255,12 @@
                             </form>
                         </div>
                     </div>
+                    {{with .Paused}}
+                    <div class="text-xs text-yellow-600 mt-1">
+                        <span class="font-medium">Deliveries Paused:</span>
+                        <span>{{if .Until}}after repeated failures, until {{.Until}} ({{.Relative}}), then one waiting delivery is sent to test the target while the others wait at least one more cooldown{{else}}held while one delivery tests whether the target has recovered{{end}}</span>
+                    </div>
+                    {{end}}
                     {{range .Config}}
                     <div class="text-xs text-gray-500 break-all mt-1">
                         <span class="font-medium text-gray-700">{{.Label}}:</span>

templates/source_logs.html

@@ -16,7 +16,8 @@
         <div class="divide-y divide-gray-100">
             {{range .Events}}
             <div class="p-4" x-data="collapsible">
-                <button type="button" class="btn-small w-full flex flex-wrap justify-between gap-2 text-left" @click="toggle">
+                <!-- Not a button element: browsers do not let a button's text be selected, and an event's ID must be. -->
+                <div role="button" tabindex="0" class="btn-small w-full flex flex-wrap justify-between gap-2" :aria-expanded="open" @mousedown="cancelPendingToggle" @click="toggleUnlessSelecting" @keydown.enter.prevent="toggle" @keydown.space.prevent="toggle">
                     <span class="flex flex-wrap items-center gap-3">
                         <span class="badge-info">{{.Method}}</span>
                         <span class="text-sm font-mono text-gray-700">{{.ID}}</span>
@@ -31,15 +32,16 @@
                     <span class="flex flex-wrap items-center gap-4">
                         {{range .Deliveries}}
                         <span class="text-xs {{if eq .Status "delivered"}}text-green-600{{else if eq .Status "failed"}}text-red-600{{else if eq .Status "retrying"}}text-yellow-600{{else}}text-gray-400{{end}}">
-                            {{.Target.DisplayName}}: {{.Status}}
+                            {{.Target.DisplayName}}: {{if .Paused}}waiting{{else}}{{.Status}}{{end}}
                         </span>
                         {{end}}
                         <span class="text-xs text-gray-400">{{.CreatedAt.Format "2006-01-02 15:04:05"}}</span>
-                        <svg class="w-4 h-4 text-gray-400 transition-transform" :class="caretClass" fill="none" stroke="currentColor" viewBox="0 0 24 24">
+                        <!-- The caret has no text to select, so a click on it toggles at once. -->
+                        <svg class="w-4 h-4 text-gray-400 transition-transform" :class="caretClass" @click.stop="toggle" fill="none" stroke="currentColor" viewBox="0 0 24 24">
                             <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M19 9l-7 7-7-7"/>
                         </svg>
                     </span>
-                </button>
+                </div>
 
                 <div x-show="open" x-cloak class="mt-3 p-3 bg-gray-50 rounded-md">
                     <div class="mb-3 flex flex-wrap items-center justify-between gap-2">
@@ -65,7 +67,7 @@
                                     <button type="button" class="btn-small flex-1 flex-wrap justify-between gap-2 text-left" @click="toggle">
                                         <span class="flex flex-wrap items-center gap-3">
                                             <span class="text-sm text-gray-700">{{.Target.DisplayName}}</span>
-                                            <span class="text-xs {{if eq .Status "delivered"}}text-green-600{{else if eq .Status "failed"}}text-red-600{{else if eq .Status "retrying"}}text-yellow-600{{else}}text-gray-400{{end}}">{{.Status}}</span>
+                                            <span class="text-xs {{if eq .Status "delivered"}}text-green-600{{else if eq .Status "failed"}}text-red-600{{else if eq .Status "retrying"}}text-yellow-600{{else}}text-gray-400{{end}}">{{with .Paused}}waiting: target paused after repeated failures, next try no earlier than {{.Until}} ({{.Relative}}){{else}}{{.Status}}{{end}}</span>
                                         </span>
                                         <span class="flex flex-wrap items-center gap-3">
                                             <span class="text-xs text-gray-400">{{.AttemptCount}} attempt{{if ne .AttemptCount 1}}s{{end}}</span>
@@ -74,7 +76,7 @@
                                             </svg>
                                         </span>
                                     </button>
-                                    {{if .Status.Terminal}}
+                                    {{if and .Status.Terminal (not .Target.Deleted)}}
                                     <form method="POST" action="/hook/{{$.Webhook.ID}}/deliveries/{{.ID}}/replay" class="inline">
                                         <input type="hidden" name="csrf_token" value="{{$.CSRFToken}}">
                                         <input type="hidden" name="page" value="{{$.Page}}">