Show a target paused by its circuit breaker (closes #385)

While an http or slack target's circuit breaker is open, the target's
row on the webhook page says its deliveries are paused until the
cooldown ends, in UTC and from now. Each of its retrying deliveries
shows as waiting in the event log and on the event's page, until the
later of the cooldown's end and the end of its own backoff. While the
breaker is half-open, the row says deliveries are held while one
delivery tests the target, with no time, and deliveries keep their
plain status.

The engine gains one read, StateAndCooldown(targetID), taking a
breaker's state and remaining cooldown under one lock; the handlers
reach it through a one-method interface wired like Archives.

Model: opus-5-5

README.md

@@ -1699,6 +1699,7 @@ events should be forwarded.
 | `active`         | boolean    | Whether deliveries are enabled (default: true) |
 | `config`         | JSON text  | Type-specific configuration |
 | `max_retries`    | integer    | Total delivery attempts for `http` and `slack` targets, not retries on top of the first: 0 is a single fire-and-forget attempt with no retries and no circuit breaker, and a value of N makes N attempts in all, with exponential backoff and a per-target circuit breaker. Ignored by `database` and `log` targets |
+| `max_queue_size` | integer    | Stored and shown on the target's detail view, but not enforced anywhere yet: nothing in the delivery engine consults it. Queue depth is set by the two fixed 10,000-entry channels |
 
 **Relations:** Belongs to Webhook. Has many Deliveries.
 
@@ -2369,16 +2370,12 @@ the breaker turns it away.
 While a target's breaker is open, the target's row on the webhook page
 says its deliveries are paused until the cooldown ends, in UTC and as a
 time from now. Each of its `retrying` deliveries shows as waiting in the
-event log and on the event's page, with the earliest it can be tried
-next: the later of the cooldown's end and the end of its own backoff
-after its last attempt. It is only the earliest: when the cooldown ends,
-one of the target's waiting deliveries is sent to test it while the
-others wait at least one more cooldown, as the row also says. A time not
-on the current UTC day is shown with its date. While the breaker is
-half-open, the row says instead that deliveries are held while one
-delivery tests whether the target has recovered, with no time, and the
-target's deliveries show their plain status, since any of them may be
-the one being sent.
+event log and on the event's page, with the time it will be tried next:
+the later of the cooldown's end and the end of its own backoff after its
+last attempt. While the breaker is half-open, the row says instead that
+deliveries are held while one delivery tests whether the target has
+recovered, with no time, and the target's deliveries show their plain
+status, since any of them may be the one being sent.
 
 ### Metrics
 

internal/database/model_target.go

@@ -31,7 +31,8 @@ type Target struct {
 
 	// For HTTP targets (max_retries=0 means fire-and-forget,
 	// >0 enables retries with backoff)
-	MaxRetries int `json:"maxRetries,omitempty"`
+	MaxRetries   int `json:"maxRetries,omitempty"`
+	MaxQueueSize int `json:"maxQueueSize,omitempty"`
 
 	// Relations. No model marshals the record it belongs to:
 	// Webhook.Targets leads back here, and the JSON could loop.

internal/delivery/engine.go

@@ -311,11 +311,13 @@ func (e *Engine) Rename(
 	return e.dbTarget.rename(targetID, webhookName, targetName)
 }
 
-// StateAndCooldown implements CircuitBreakers. It returns the state of
-// the target's circuit breaker and, while the breaker is open, what is
-// left of its cooldown; the cooldown is zero once that has passed and
-// in any other state. A target with no breaker reads as closed with no
-// cooldown, and reading never creates one.
+// StateAndCooldown implements CircuitBreakers. It is
+// CircuitBreaker.StateAndCooldown for the target's breaker. While the
+// breaker is open, the pages show the target's deliveries as paused
+// until its cooldown ends; while it is half-open, they show them as
+// held, with no time, while one delivery tests whether the target has
+// recovered. A target with no breaker reads as closed, and reading
+// never creates one.
 func (e *Engine) StateAndCooldown(
 	targetID string,
 ) (CircuitState, time.Duration) {

internal/delivery/target_config_view.go

@@ -173,6 +173,13 @@ func httpConfigFields(t *database.Target) []ConfigField {
 
 	fields = append(fields, maxRetriesField(t))
 
+	if t.MaxQueueSize > 0 {
+		fields = append(fields, ConfigField{
+			Label: "Max Queue Size",
+			Value: strconv.Itoa(t.MaxQueueSize),
+		})
+	}
+
 	return fields
 }
 

internal/delivery/target_config_view_test.go

@@ -197,12 +197,14 @@ func TestNewTargetViews_Slack(t *testing.T) {
 }
 
 // TestNewTargetViews_SlackRetries proves a Slack target shows
-// its retry count the same way an HTTP target does.
+// its retry count the same way an HTTP target does, and no
+// queue size even when one is stored: delivery never reads it.
 func TestNewTargetViews_SlackRetries(t *testing.T) {
 	t.Parallel()
 
 	target := slackTarget()
 	target.MaxRetries = 2
+	target.MaxQueueSize = 100
 
 	view := viewFor(t, target)
 
@@ -224,7 +226,8 @@ func TestNewTargetViews_HTTP(t *testing.T) {
 		Config: `{"url":"` + viewExampleHook + `",` +
 			`"timeout":30,` +
 			`"headers":{"Authorization":"Bearer sekrit"}}`,
-		MaxRetries: 5,
+		MaxRetries:   5,
+		MaxQueueSize: 100,
 	})
 
 	fields := fieldMap(view.Config)
@@ -236,6 +239,7 @@ func TestNewTargetViews_HTTP(t *testing.T) {
 			"Timeout":         "30s",
 			"Headers":         "1 configured",
 			viewMaxRetries:    "5",
+			"Max Queue Size":  "100",
 		},
 		fields,
 	)

internal/handlers/handlers.go

@@ -112,25 +112,22 @@ type Handlers struct {
 // parsePageTemplate parses a page-specific template set from the
 // embedded FS. Each page template is combined with the shared
 // base, htmlheader, navbar and notice templates, and with any further
-// files the page includes. The set is named after the page file, so
-// the page's root action ({{template "base" .}}) is its entry point.
-//
-// The page file is parsed last because a later definition of a name
-// replaces an earlier one: the page's {{define "title"}} must replace
-// the {{block "title"}} fallback in htmlheader.html.
+// files the page includes. The page file must be listed first so that
+// its root action ({{template "base" .}}) becomes the template set's
+// entry point.
 func parsePageTemplate(
 	pageFile string, included ...string,
 ) *template.Template {
 	files := append([]string{
+		pageFile,
 		"base.html",
 		"htmlheader.html",
 		"navbar.html",
 		"notice.html",
 	}, included...)
-	files = append(files, pageFile)
 
 	return template.Must(
-		template.New(pageFile).ParseFS(templates.Templates, files...),
+		template.ParseFS(templates.Templates, files...),
 	)
 }
 

internal/handlers/page_title_test.go

@@ -1,107 +0,0 @@
-package handlers_test
-
-import (
-	"html/template"
-	"net/http"
-	"strings"
-	"testing"
-
-	"github.com/stretchr/testify/assert"
-	"github.com/stretchr/testify/require"
-	"sneak.berlin/go/webhooker/internal/database"
-	"sneak.berlin/go/webhooker/internal/handlers"
-	"sneak.berlin/go/webhooker/internal/session"
-	"sneak.berlin/go/webhooker/templates"
-)
-
-// TestEveryPageRendersItsOwnTitle renders each page template and checks
-// the browser tab title is the one the page declares, not the
-// "Webhooker" fallback in htmlheader.html. A page that fails to render
-// shows the error page's title instead, and fails here too.
-func TestEveryPageRendersItsOwnTitle(t *testing.T) {
-	t.Parallel()
-
-	var h *handlers.Handlers
-
-	var sess *session.Session
-
-	app := newTestApp(t, &h, &sess)
-	app.RequireStart()
-
-	t.Cleanup(app.RequireStop)
-
-	// A pointer, as in the handlers: some pages call
-	// Webhook.RetentionLabel, a pointer method.
-	webhook := &database.Webhook{Name: "orders", RetentionDays: 14}
-	webhook.ID = testWebhookID
-
-	pages := []struct {
-		page  string
-		data  map[string]any
-		title string
-	}{
-		{"login.html", map[string]any{}, "Login - Webhooker"},
-		{"profile.html", map[string]any{}, "Profile - Webhooker"},
-		{"settings.html", map[string]any{}, "Settings - Webhooker"},
-		{"sources_list.html", map[string]any{}, "Webhooks - Webhooker"},
-		{"sources_new.html", map[string]any{}, "New Webhook - Webhooker"},
-		{
-			"source_detail.html",
-			map[string]any{dataKeyWebhook: webhook},
-			"orders - Webhooker",
-		},
-		{
-			"source_edit.html",
-			map[string]any{dataKeyWebhook: webhook},
-			"Edit orders - Webhooker",
-		},
-		{
-			"source_logs.html",
-			map[string]any{dataKeyWebhook: webhook, "TotalEvents": int64(0)},
-			"Full Event Log - orders - Webhooker",
-		},
-		{
-			"event_detail.html",
-			map[string]any{dataKeyWebhook: webhook},
-			"Event - orders - Webhooker",
-		},
-		{
-			"target_edit.html",
-			map[string]any{
-				dataKeyWebhook: webhook,
-				"Target":       map[string]any{"Name": "alerts", "Type": "slack"},
-			},
-			"Edit alerts - Webhooker",
-		},
-		{
-			"error.html",
-			map[string]any{"StatusText": http.StatusText(http.StatusNotFound)},
-			"Not Found - Webhooker",
-		},
-	}
-
-	for _, p := range pages {
-		body := renderPage(t, h, sess, p.page, p.data)
-
-		_, afterOpen, _ := strings.Cut(body, "<title>")
-		title, _, _ := strings.Cut(afterOpen, "</title>")
-
-		assert.Equal(t, p.title, title, p.page)
-	}
-}
-
-// TestTitleFallbackIsWebhooker checks the title htmlheader.html gives a
-// page that declares none. Every page declares one, so it is checked on
-// htmlheader.html alone.
-func TestTitleFallbackIsWebhooker(t *testing.T) {
-	t.Parallel()
-
-	header := template.Must(
-		template.ParseFS(templates.Templates, "htmlheader.html"),
-	)
-
-	var buf strings.Builder
-
-	require.NoError(t, header.ExecuteTemplate(&buf, "htmlheader", nil))
-	assert.Contains(t, buf.String(), "<title>Webhooker</title>")
-}

internal/handlers/target_list.go

@@ -31,12 +31,10 @@ type TargetRowView struct {
 }
 
 // PausedView is a target's circuit breaker turning deliveries away.
-// While the breaker is open, Until is a time in UTC, and Relative how
-// long that is from now: on the target's row, when the cooldown ends;
-// on a delivery, the earliest it can be tried next. While it is
-// half-open both are empty: the cooldown has ended, and the target's
-// deliveries are held while one delivery tests whether the target has
-// recovered.
+// While the breaker is open, Until is when they resume, in UTC, and
+// Relative how long that is from now. While it is half-open both are
+// empty: the cooldown has ended, and the target's deliveries are held
+// while one delivery tests whether the target has recovered.
 type PausedView struct {
 	Until    string
 	Relative string
@@ -58,13 +56,11 @@ func (h *Handlers) pausedView(targetID string) *PausedView {
 }
 
 // deliveryPausedView reads the circuit breaker of a retrying delivery's
-// target. While it is open, it says the earliest the delivery can be
-// tried next: the later of the cooldown's end and the end of the
-// delivery's own backoff after its last attempt. It is only the
-// earliest: when the cooldown ends, one of the target's waiting
-// deliveries is sent to test it while the others wait at least one more
-// cooldown. Otherwise it returns nil, half-open included, since the
-// delivery may then be the one being sent to test the target.
+// target. While it is open, it says when the delivery will be tried
+// next: the later of the cooldown's end and the end of the delivery's
+// own backoff after its last attempt. Otherwise it returns nil, half-open
+// included, since the delivery may then be the one being sent to test
+// the target.
 func (h *Handlers) deliveryPausedView(
 	targetID string, attempts []deliveryResultRow,
 ) *PausedView {
@@ -87,19 +83,10 @@ func (h *Handlers) deliveryPausedView(
 	return newPausedView(next)
 }
 
-// newPausedView is a PausedView of deliveries paused until the given
-// time. A time not on the current UTC day is written with its date, as
-// the event log writes its times.
+// newPausedView is a PausedView of deliveries that resume at until.
 func newPausedView(until time.Time) *PausedView {
-	until = until.UTC()
-
-	layout := time.TimeOnly
-	if until.Format(time.DateOnly) != time.Now().UTC().Format(time.DateOnly) {
-		layout = time.DateTime
-	}
-
 	return &PausedView{
-		Until:    until.Format(layout) + " UTC",
+		Until:    until.UTC().Format(time.TimeOnly) + " UTC",
 		Relative: humanize.Time(until),
 	}
 }

internal/handlers/target_paused_test.go

@@ -15,24 +15,22 @@ import (
 	"sneak.berlin/go/webhooker/internal/session"
 )
 
-// cooldownEnds is how the pages write the end of a paused target's
-// breaker's cooldown: the time in UTC, with its date when that falls on
-// another UTC day, then how long that is from now.
-const cooldownEnds = `(\d{4}-\d\d-\d\d )?\d\d:\d\d:\d\d UTC ` +
-	`\(\d+ seconds from now\)`
+// resumesAt is how the pages write when a paused target's deliveries
+// resume at the end of its breaker's cooldown: the time in UTC, then
+// how long that is from now.
+const resumesAt = `\d\d:\d\d:\d\d UTC \(\d+ seconds from now\)`
 
 // TestPausedTarget_ShownUntilBreakerCloses takes an http target's
 // circuit breaker from open through half-open to closed.
 //
 // Open, the target's row on the webhook page says its deliveries are
 // paused and until when, and each retrying delivery says it is waiting
-// and why in the event log and on the event's page, with the earliest
-// it can be tried next: the later of the cooldown's end and the end of
-// its own backoff, with the date when that is another UTC day.
-// Half-open, the row says deliveries are held while one delivery tests
-// the target, with no time, and no delivery says it is waiting. Closed,
-// the pages say neither. The delivered delivery and the log target are
-// shown as before throughout.
+// and why in the event log and on the event's page, until the later of
+// the cooldown's end and the end of its own backoff. Half-open, the row
+// says deliveries are held while one delivery tests the target, with no
+// time, and no delivery says it is waiting. Closed, the pages say
+// neither. The delivered delivery and the log target are shown as
+// before throughout.
 func TestPausedTarget_ShownUntilBreakerCloses(t *testing.T) {
 	t.Parallel()
 
@@ -61,40 +59,38 @@ func TestPausedTarget_ShownUntilBreakerCloses(t *testing.T) {
 	addDelivery(t, dbMgr, wh.ID, delivered.ID, target.ID,
 		database.DeliveryStatusDelivered)
 
-	// This delivery's 18th attempt failed a minute ago, so its own
-	// backoff ends over a day from now: long after the cooldown, and on
-	// another UTC day, so the page shows the date.
+	// This delivery's 13th attempt failed a minute ago, so its own
+	// backoff ends over an hour from now, long after the cooldown.
 	backedOff := seedStoredEvent(t, dbMgr, wh.ID, `{"n":3}`)
 	backedOffID := addDelivery(t, dbMgr, wh.ID, backedOff.ID, target.ID,
 		database.DeliveryStatusRetrying)
 
 	failedAt := time.Now().Add(-time.Minute).Truncate(time.Second)
-	addFailedAttempt(t, dbMgr, wh.ID, backedOffID, 18, failedAt)
+	addFailedAttempt(t, dbMgr, wh.ID, backedOffID, 13, failedAt)
 
-	backoffEnds := failedAt.Add(delivery.Backoff(18)).UTC().
-		Format("2006-01-02 15:04:05") + " UTC (1 day from now)"
+	backoffEnds := failedAt.Add(delivery.Backoff(13)).UTC().
+		Format(time.TimeOnly) + " UTC (1 hour from now)"
 
 	const waiting = "waiting: target paused after repeated failures, " +
-		"next try no earlier than "
+		"resumes "
 
 	breakers.Set(target.ID, delivery.CircuitOpen, 30*time.Second)
 
 	list := targetList(t, renderSourceDetailPage(t, h, sess, wh.ID))
 	assert.Regexp(t, "t-http http Active Edit Deactivate Delete "+
-		"Deliveries Paused: after repeated failures, until "+cooldownEnds+
-		", then one waiting delivery is sent to test the target while "+
-		"the others wait at least one more cooldown", list)
+		"Deliveries Paused: after repeated failures, until "+resumesAt,
+		list)
 	assert.Equal(t, 1, strings.Count(list, "Paused"))
 
 	log := renderSourceLogsPage(t, h, sess, wh.ID)
 	assert.Equal(t, 2, strings.Count(log, "t-http: waiting"))
 	assert.Contains(t, log, "t-http: delivered")
-	assert.Regexp(t, waiting+cooldownEnds, log)
+	assert.Regexp(t, waiting+resumesAt, log)
 	assert.Contains(t, log, waiting+backoffEnds)
 	assert.NotContains(t, log, "retrying")
 
 	page := eventPage(t, h, sess, wh.ID, retrying.ID)
-	assert.Regexp(t, waiting+cooldownEnds, page)
+	assert.Regexp(t, waiting+resumesAt, page)
 	assert.NotContains(t, page, "retrying")
 
 	page = eventPage(t, h, sess, wh.ID, backedOff.ID)

templates/event_detail.html

@@ -69,7 +69,7 @@
                 <div class="flex flex-wrap items-center justify-between gap-3">
                     <span class="text-sm text-gray-700">{{.Target.DisplayName}}</span>
                     <span class="flex flex-wrap items-center gap-3">
-                        <span class="text-xs {{if eq .Status "delivered"}}text-green-600{{else if eq .Status "failed"}}text-red-600{{else if eq .Status "retrying"}}text-yellow-600{{else}}text-gray-400{{end}}">{{with .Paused}}waiting: target paused after repeated failures, next try no earlier than {{.Until}} ({{.Relative}}){{else}}{{.Status}}{{end}}</span>
+                        <span class="text-xs {{if eq .Status "delivered"}}text-green-600{{else if eq .Status "failed"}}text-red-600{{else if eq .Status "retrying"}}text-yellow-600{{else}}text-gray-400{{end}}">{{with .Paused}}waiting: target paused after repeated failures, resumes {{.Until}} ({{.Relative}}){{else}}{{.Status}}{{end}}</span>
                         <span class="text-xs text-gray-400">{{.AttemptCount}} attempt{{if ne .AttemptCount 1}}s{{end}}</span>
                     </span>
                 </div>

templates/source_detail.html

@@ -258,7 +258,7 @@
                     {{with .Paused}}
                     <div class="text-xs text-yellow-600 mt-1">
                         <span class="font-medium">Deliveries Paused:</span>
-                        <span>{{if .Until}}after repeated failures, until {{.Until}} ({{.Relative}}), then one waiting delivery is sent to test the target while the others wait at least one more cooldown{{else}}held while one delivery tests whether the target has recovered{{end}}</span>
+                        <span>{{if .Until}}after repeated failures, until {{.Until}} ({{.Relative}}){{else}}held while one delivery tests whether the target has recovered{{end}}</span>
                     </div>
                     {{end}}
                     {{range .Config}}

templates/source_logs.html

@@ -65,7 +65,7 @@
                                     <button type="button" class="btn-small flex-1 flex-wrap justify-between gap-2 text-left" @click="toggle">
                                         <span class="flex flex-wrap items-center gap-3">
                                             <span class="text-sm text-gray-700">{{.Target.DisplayName}}</span>
-                                            <span class="text-xs {{if eq .Status "delivered"}}text-green-600{{else if eq .Status "failed"}}text-red-600{{else if eq .Status "retrying"}}text-yellow-600{{else}}text-gray-400{{end}}">{{with .Paused}}waiting: target paused after repeated failures, next try no earlier than {{.Until}} ({{.Relative}}){{else}}{{.Status}}{{end}}</span>
+                                            <span class="text-xs {{if eq .Status "delivered"}}text-green-600{{else if eq .Status "failed"}}text-red-600{{else if eq .Status "retrying"}}text-yellow-600{{else}}text-gray-400{{end}}">{{with .Paused}}waiting: target paused after repeated failures, resumes {{.Until}} ({{.Relative}}){{else}}{{.Status}}{{end}}</span>
                                         </span>
                                         <span class="flex flex-wrap items-center gap-3">
                                             <span class="text-xs text-gray-400">{{.AttemptCount}} attempt{{if ne .AttemptCount 1}}s{{end}}</span>