Show a target paused by its circuit breaker (closes #385)

While an http or slack target's circuit breaker is open, the target's
row on the webhook page says its deliveries are paused until the
cooldown ends, in UTC and from now. Each of its retrying deliveries
shows as waiting in the event log and on the event's page, until the
later of the cooldown's end and the end of its own backoff. While the
breaker is half-open, the row says deliveries are held while one
delivery tests the target, with no time, and deliveries keep their
plain status.

The engine gains one read, StateAndCooldown(targetID), taking a
breaker's state and remaining cooldown under one lock; the handlers
reach it through a one-method interface wired like Archives.

Model: opus-5-5

README.md

@@ -1699,6 +1699,7 @@ events should be forwarded.
 | `active`         | boolean    | Whether deliveries are enabled (default: true) |
 | `config`         | JSON text  | Type-specific configuration |
 | `max_retries`    | integer    | Total delivery attempts for `http` and `slack` targets, not retries on top of the first: 0 is a single fire-and-forget attempt with no retries and no circuit breaker, and a value of N makes N attempts in all, with exponential backoff and a per-target circuit breaker. Ignored by `database` and `log` targets |
+| `max_queue_size` | integer    | Stored and shown on the target's detail view, but not enforced anywhere yet: nothing in the delivery engine consults it. Queue depth is set by the two fixed 10,000-entry channels |
 
 **Relations:** Belongs to Webhook. Has many Deliveries.
 
@@ -2369,16 +2370,12 @@ the breaker turns it away.
 While a target's breaker is open, the target's row on the webhook page
 says its deliveries are paused until the cooldown ends, in UTC and as a
 time from now. Each of its `retrying` deliveries shows as waiting in the
-event log and on the event's page, with the earliest it can be tried
+event log and on the event's page, with the time it will be tried next:
-next: the later of the cooldown's end and the end of its own backoff
+the later of the cooldown's end and the end of its own backoff after its
-after its last attempt. It is only the earliest: when the cooldown ends,
+last attempt. While the breaker is half-open, the row says instead that
-one of the target's waiting deliveries is sent to test it while the
+deliveries are held while one delivery tests whether the target has
-others wait at least one more cooldown, as the row also says. A time not
+recovered, with no time, and the target's deliveries show their plain
-on the current UTC day is shown with its date. While the breaker is
+status, since any of them may be the one being sent.
-half-open, the row says instead that deliveries are held while one
-delivery tests whether the target has recovered, with no time, and the
-target's deliveries show their plain status, since any of them may be
-the one being sent.
 
 ### Metrics
 

internal/database/model_target.go

@@ -32,6 +32,7 @@ type Target struct {
 	// For HTTP targets (max_retries=0 means fire-and-forget,
 	// >0 enables retries with backoff)
 	MaxRetries   int `json:"maxRetries,omitempty"`
+	MaxQueueSize int `json:"maxQueueSize,omitempty"`
 
 	// Relations. No model marshals the record it belongs to:
 	// Webhook.Targets leads back here, and the JSON could loop.

internal/delivery/engine.go

@@ -311,11 +311,13 @@ func (e *Engine) Rename(
 	return e.dbTarget.rename(targetID, webhookName, targetName)
 }
 
-// StateAndCooldown implements CircuitBreakers. It returns the state of
+// StateAndCooldown implements CircuitBreakers. It is
-// the target's circuit breaker and, while the breaker is open, what is
+// CircuitBreaker.StateAndCooldown for the target's breaker. While the
-// left of its cooldown; the cooldown is zero once that has passed and
+// breaker is open, the pages show the target's deliveries as paused
-// in any other state. A target with no breaker reads as closed with no
+// until its cooldown ends; while it is half-open, they show them as
-// cooldown, and reading never creates one.
+// held, with no time, while one delivery tests whether the target has
+// recovered. A target with no breaker reads as closed, and reading
+// never creates one.
 func (e *Engine) StateAndCooldown(
 	targetID string,
 ) (CircuitState, time.Duration) {

internal/delivery/target_config_view.go

@@ -173,6 +173,13 @@ func httpConfigFields(t *database.Target) []ConfigField {
 
 	fields = append(fields, maxRetriesField(t))
 
+	if t.MaxQueueSize > 0 {
+		fields = append(fields, ConfigField{
+			Label: "Max Queue Size",
+			Value: strconv.Itoa(t.MaxQueueSize),
+		})
+	}
+
 	return fields
 }
 

internal/delivery/target_config_view_test.go

@@ -197,12 +197,14 @@ func TestNewTargetViews_Slack(t *testing.T) {
 }
 
 // TestNewTargetViews_SlackRetries proves a Slack target shows
-// its retry count the same way an HTTP target does.
+// its retry count the same way an HTTP target does, and no
+// queue size even when one is stored: delivery never reads it.
 func TestNewTargetViews_SlackRetries(t *testing.T) {
 	t.Parallel()
 
 	target := slackTarget()
 	target.MaxRetries = 2
+	target.MaxQueueSize = 100
 
 	view := viewFor(t, target)
 
@@ -225,6 +227,7 @@ func TestNewTargetViews_HTTP(t *testing.T) {
 			`"timeout":30,` +
 			`"headers":{"Authorization":"Bearer sekrit"}}`,
 		MaxRetries:   5,
+		MaxQueueSize: 100,
 	})
 
 	fields := fieldMap(view.Config)
@@ -236,6 +239,7 @@ func TestNewTargetViews_HTTP(t *testing.T) {
 			"Timeout":         "30s",
 			"Headers":         "1 configured",
 			viewMaxRetries:    "5",
+			"Max Queue Size":  "100",
 		},
 		fields,
 	)

internal/handlers/handlers.go

@@ -112,25 +112,22 @@ type Handlers struct {
 // parsePageTemplate parses a page-specific template set from the
 // embedded FS. Each page template is combined with the shared
 // base, htmlheader, navbar and notice templates, and with any further
-// files the page includes. The set is named after the page file, so
+// files the page includes. The page file must be listed first so that
-// the page's root action ({{template "base" .}}) is its entry point.
+// its root action ({{template "base" .}}) becomes the template set's
-//
+// entry point.
-// The page file is parsed last because a later definition of a name
-// replaces an earlier one: the page's {{define "title"}} must replace
-// the {{block "title"}} fallback in htmlheader.html.
 func parsePageTemplate(
 	pageFile string, included ...string,
 ) *template.Template {
 	files := append([]string{
+		pageFile,
 		"base.html",
 		"htmlheader.html",
 		"navbar.html",
 		"notice.html",
 	}, included...)
-	files = append(files, pageFile)
 
 	return template.Must(
-		template.New(pageFile).ParseFS(templates.Templates, files...),
+		template.ParseFS(templates.Templates, files...),
 	)
 }
 

internal/handlers/page_title_test.go

@@ -1,107 +0,0 @@
-package handlers_test
-
-import (
-	"html/template"
-	"net/http"
-	"strings"
-	"testing"
-
-	"github.com/stretchr/testify/assert"
-	"github.com/stretchr/testify/require"
-	"sneak.berlin/go/webhooker/internal/database"
-	"sneak.berlin/go/webhooker/internal/handlers"
-	"sneak.berlin/go/webhooker/internal/session"
-	"sneak.berlin/go/webhooker/templates"
-)
-
-// TestEveryPageRendersItsOwnTitle renders each page template and checks
-// the browser tab title is the one the page declares, not the
-// "Webhooker" fallback in htmlheader.html. A page that fails to render
-// shows the error page's title instead, and fails here too.
-func TestEveryPageRendersItsOwnTitle(t *testing.T) {
-	t.Parallel()
-
-	var h *handlers.Handlers
-
-	var sess *session.Session
-
-	app := newTestApp(t, &h, &sess)
-	app.RequireStart()
-
-	t.Cleanup(app.RequireStop)
-
-	// A pointer, as in the handlers: some pages call
-	// Webhook.RetentionLabel, a pointer method.
-	webhook := &database.Webhook{Name: "orders", RetentionDays: 14}
-	webhook.ID = testWebhookID
-
-	pages := []struct {
-		page  string
-		data  map[string]any
-		title string
-	}{
-		{"login.html", map[string]any{}, "Login - Webhooker"},
-		{"profile.html", map[string]any{}, "Profile - Webhooker"},
-		{"settings.html", map[string]any{}, "Settings - Webhooker"},
-		{"sources_list.html", map[string]any{}, "Webhooks - Webhooker"},
-		{"sources_new.html", map[string]any{}, "New Webhook - Webhooker"},
-		{
-			"source_detail.html",
-			map[string]any{dataKeyWebhook: webhook},
-			"orders - Webhooker",
-		},
-		{
-			"source_edit.html",
-			map[string]any{dataKeyWebhook: webhook},
-			"Edit orders - Webhooker",
-		},
-		{
-			"source_logs.html",
-			map[string]any{dataKeyWebhook: webhook, "TotalEvents": int64(0)},
-			"Full Event Log - orders - Webhooker",
-		},
-		{
-			"event_detail.html",
-			map[string]any{dataKeyWebhook: webhook},
-			"Event - orders - Webhooker",
-		},
-		{
-			"target_edit.html",
-			map[string]any{
-				dataKeyWebhook: webhook,
-				"Target":       map[string]any{"Name": "alerts", "Type": "slack"},
-			},
-			"Edit alerts - Webhooker",
-		},
-		{
-			"error.html",
-			map[string]any{"StatusText": http.StatusText(http.StatusNotFound)},
-			"Not Found - Webhooker",
-		},
-	}
-
-	for _, p := range pages {
-		body := renderPage(t, h, sess, p.page, p.data)
-
-		_, afterOpen, _ := strings.Cut(body, "<title>")
-		title, _, _ := strings.Cut(afterOpen, "</title>")
-
-		assert.Equal(t, p.title, title, p.page)
-	}
-}
-
-// TestTitleFallbackIsWebhooker checks the title htmlheader.html gives a
-// page that declares none. Every page declares one, so it is checked on
-// htmlheader.html alone.
-func TestTitleFallbackIsWebhooker(t *testing.T) {
-	t.Parallel()
-
-	header := template.Must(
-		template.ParseFS(templates.Templates, "htmlheader.html"),
-	)
-
-	var buf strings.Builder
-
-	require.NoError(t, header.ExecuteTemplate(&buf, "htmlheader", nil))
-	assert.Contains(t, buf.String(), "<title>Webhooker</title>")
-}

internal/handlers/target_list.go

@@ -31,12 +31,10 @@ type TargetRowView struct {
 }
 
 // PausedView is a target's circuit breaker turning deliveries away.
-// While the breaker is open, Until is a time in UTC, and Relative how
+// While the breaker is open, Until is when they resume, in UTC, and
-// long that is from now: on the target's row, when the cooldown ends;
+// Relative how long that is from now. While it is half-open both are
-// on a delivery, the earliest it can be tried next. While it is
+// empty: the cooldown has ended, and the target's deliveries are held
-// half-open both are empty: the cooldown has ended, and the target's
+// while one delivery tests whether the target has recovered.
-// deliveries are held while one delivery tests whether the target has
-// recovered.
 type PausedView struct {
 	Until    string
 	Relative string
@@ -58,13 +56,11 @@ func (h *Handlers) pausedView(targetID string) *PausedView {
 }
 
 // deliveryPausedView reads the circuit breaker of a retrying delivery's
-// target. While it is open, it says the earliest the delivery can be
+// target. While it is open, it says when the delivery will be tried
-// tried next: the later of the cooldown's end and the end of the
+// next: the later of the cooldown's end and the end of the delivery's
-// delivery's own backoff after its last attempt. It is only the
+// own backoff after its last attempt. Otherwise it returns nil, half-open
-// earliest: when the cooldown ends, one of the target's waiting
+// included, since the delivery may then be the one being sent to test
-// deliveries is sent to test it while the others wait at least one more
+// the target.
-// cooldown. Otherwise it returns nil, half-open included, since the
-// delivery may then be the one being sent to test the target.
 func (h *Handlers) deliveryPausedView(
 	targetID string, attempts []deliveryResultRow,
 ) *PausedView {
@@ -87,19 +83,10 @@ func (h *Handlers) deliveryPausedView(
 	return newPausedView(next)
 }
 
-// newPausedView is a PausedView of deliveries paused until the given
+// newPausedView is a PausedView of deliveries that resume at until.
-// time. A time not on the current UTC day is written with its date, as
-// the event log writes its times.
 func newPausedView(until time.Time) *PausedView {
-	until = until.UTC()
-
-	layout := time.TimeOnly
-	if until.Format(time.DateOnly) != time.Now().UTC().Format(time.DateOnly) {
-		layout = time.DateTime
-	}
-
 	return &PausedView{
-		Until:    until.Format(layout) + " UTC",
+		Until:    until.UTC().Format(time.TimeOnly) + " UTC",
 		Relative: humanize.Time(until),
 	}
 }

internal/handlers/target_paused_test.go

@@ -15,24 +15,22 @@ import (
 	"sneak.berlin/go/webhooker/internal/session"
 )
 
-// cooldownEnds is how the pages write the end of a paused target's
+// resumesAt is how the pages write when a paused target's deliveries
-// breaker's cooldown: the time in UTC, with its date when that falls on
+// resume at the end of its breaker's cooldown: the time in UTC, then
-// another UTC day, then how long that is from now.
+// how long that is from now.
-const cooldownEnds = `(\d{4}-\d\d-\d\d )?\d\d:\d\d:\d\d UTC ` +
+const resumesAt = `\d\d:\d\d:\d\d UTC \(\d+ seconds from now\)`
-	`\(\d+ seconds from now\)`
 
 // TestPausedTarget_ShownUntilBreakerCloses takes an http target's
 // circuit breaker from open through half-open to closed.
 //
 // Open, the target's row on the webhook page says its deliveries are
 // paused and until when, and each retrying delivery says it is waiting
-// and why in the event log and on the event's page, with the earliest
+// and why in the event log and on the event's page, until the later of
-// it can be tried next: the later of the cooldown's end and the end of
+// the cooldown's end and the end of its own backoff. Half-open, the row
-// its own backoff, with the date when that is another UTC day.
+// says deliveries are held while one delivery tests the target, with no
-// Half-open, the row says deliveries are held while one delivery tests
+// time, and no delivery says it is waiting. Closed, the pages say
-// the target, with no time, and no delivery says it is waiting. Closed,
+// neither. The delivered delivery and the log target are shown as
-// the pages say neither. The delivered delivery and the log target are
+// before throughout.
-// shown as before throughout.
 func TestPausedTarget_ShownUntilBreakerCloses(t *testing.T) {
 	t.Parallel()
 
@@ -61,40 +59,38 @@ func TestPausedTarget_ShownUntilBreakerCloses(t *testing.T) {
 	addDelivery(t, dbMgr, wh.ID, delivered.ID, target.ID,
 		database.DeliveryStatusDelivered)
 
-	// This delivery's 18th attempt failed a minute ago, so its own
+	// This delivery's 13th attempt failed a minute ago, so its own
-	// backoff ends over a day from now: long after the cooldown, and on
+	// backoff ends over an hour from now, long after the cooldown.
-	// another UTC day, so the page shows the date.
 	backedOff := seedStoredEvent(t, dbMgr, wh.ID, `{"n":3}`)
 	backedOffID := addDelivery(t, dbMgr, wh.ID, backedOff.ID, target.ID,
 		database.DeliveryStatusRetrying)
 
 	failedAt := time.Now().Add(-time.Minute).Truncate(time.Second)
-	addFailedAttempt(t, dbMgr, wh.ID, backedOffID, 18, failedAt)
+	addFailedAttempt(t, dbMgr, wh.ID, backedOffID, 13, failedAt)
 
-	backoffEnds := failedAt.Add(delivery.Backoff(18)).UTC().
+	backoffEnds := failedAt.Add(delivery.Backoff(13)).UTC().
-		Format("2006-01-02 15:04:05") + " UTC (1 day from now)"
+		Format(time.TimeOnly) + " UTC (1 hour from now)"
 
 	const waiting = "waiting: target paused after repeated failures, " +
-		"next try no earlier than "
+		"resumes "
 
 	breakers.Set(target.ID, delivery.CircuitOpen, 30*time.Second)
 
 	list := targetList(t, renderSourceDetailPage(t, h, sess, wh.ID))
 	assert.Regexp(t, "t-http http Active Edit Deactivate Delete "+
-		"Deliveries Paused: after repeated failures, until "+cooldownEnds+
+		"Deliveries Paused: after repeated failures, until "+resumesAt,
-		", then one waiting delivery is sent to test the target while "+
+		list)
-		"the others wait at least one more cooldown", list)
 	assert.Equal(t, 1, strings.Count(list, "Paused"))
 
 	log := renderSourceLogsPage(t, h, sess, wh.ID)
 	assert.Equal(t, 2, strings.Count(log, "t-http: waiting"))
 	assert.Contains(t, log, "t-http: delivered")
-	assert.Regexp(t, waiting+cooldownEnds, log)
+	assert.Regexp(t, waiting+resumesAt, log)
 	assert.Contains(t, log, waiting+backoffEnds)
 	assert.NotContains(t, log, "retrying")
 
 	page := eventPage(t, h, sess, wh.ID, retrying.ID)
-	assert.Regexp(t, waiting+cooldownEnds, page)
+	assert.Regexp(t, waiting+resumesAt, page)
 	assert.NotContains(t, page, "retrying")
 
 	page = eventPage(t, h, sess, wh.ID, backedOff.ID)

templates/event_detail.html

@@ -69,7 +69,7 @@
                 <div class="flex flex-wrap items-center justify-between gap-3">
                     <span class="text-sm text-gray-700">{{.Target.DisplayName}}</span>
                     <span class="flex flex-wrap items-center gap-3">
-                        <span class="text-xs {{if eq .Status "delivered"}}text-green-600{{else if eq .Status "failed"}}text-red-600{{else if eq .Status "retrying"}}text-yellow-600{{else}}text-gray-400{{end}}">{{with .Paused}}waiting: target paused after repeated failures, next try no earlier than {{.Until}} ({{.Relative}}){{else}}{{.Status}}{{end}}</span>
+                        <span class="text-xs {{if eq .Status "delivered"}}text-green-600{{else if eq .Status "failed"}}text-red-600{{else if eq .Status "retrying"}}text-yellow-600{{else}}text-gray-400{{end}}">{{with .Paused}}waiting: target paused after repeated failures, resumes {{.Until}} ({{.Relative}}){{else}}{{.Status}}{{end}}</span>
                         <span class="text-xs text-gray-400">{{.AttemptCount}} attempt{{if ne .AttemptCount 1}}s{{end}}</span>
                     </span>
                 </div>

templates/source_detail.html

@@ -258,7 +258,7 @@
                     {{with .Paused}}
                     <div class="text-xs text-yellow-600 mt-1">
                         <span class="font-medium">Deliveries Paused:</span>
-                        <span>{{if .Until}}after repeated failures, until {{.Until}} ({{.Relative}}), then one waiting delivery is sent to test the target while the others wait at least one more cooldown{{else}}held while one delivery tests whether the target has recovered{{end}}</span>
+                        <span>{{if .Until}}after repeated failures, until {{.Until}} ({{.Relative}}){{else}}held while one delivery tests whether the target has recovered{{end}}</span>
                     </div>
                     {{end}}
                     {{range .Config}}

templates/source_logs.html

@@ -65,7 +65,7 @@
                                     <button type="button" class="btn-small flex-1 flex-wrap justify-between gap-2 text-left" @click="toggle">
                                         <span class="flex flex-wrap items-center gap-3">
                                             <span class="text-sm text-gray-700">{{.Target.DisplayName}}</span>
-                                            <span class="text-xs {{if eq .Status "delivered"}}text-green-600{{else if eq .Status "failed"}}text-red-600{{else if eq .Status "retrying"}}text-yellow-600{{else}}text-gray-400{{end}}">{{with .Paused}}waiting: target paused after repeated failures, next try no earlier than {{.Until}} ({{.Relative}}){{else}}{{.Status}}{{end}}</span>
+                                            <span class="text-xs {{if eq .Status "delivered"}}text-green-600{{else if eq .Status "failed"}}text-red-600{{else if eq .Status "retrying"}}text-yellow-600{{else}}text-gray-400{{end}}">{{with .Paused}}waiting: target paused after repeated failures, resumes {{.Until}} ({{.Relative}}){{else}}{{.Status}}{{end}}</span>
                                         </span>
                                         <span class="flex flex-wrap items-center gap-3">
                                             <span class="text-xs text-gray-400">{{.AttemptCount}} attempt{{if ne .AttemptCount 1}}s{{end}}</span>