Show a target paused by its circuit breaker (closes #385)

While an http or slack target's circuit breaker is open, the target's
row on the webhook page says its deliveries are paused until the
cooldown ends, then one is sent to test the target. Each of its
retrying deliveries shows as waiting in the event log and on the
event's page, with the earliest it can be tried next: the later of the
cooldown's end and the end of its own backoff, dated when not today
in UTC. While the breaker is half-open, the row says deliveries are
held while one delivery tests the target, with no time, and deliveries
keep their plain status.

The engine gains one read, StateAndCooldown(targetID), taking a
breaker's state and remaining cooldown under one lock; the handlers
reach it through a one-method interface wired like Archives.

Model: opus-5-5

README.md

@@ -1699,7 +1699,6 @@ events should be forwarded.
 | `active`         | boolean    | Whether deliveries are enabled (default: true) |
 | `config`         | JSON text  | Type-specific configuration |
 | `max_retries`    | integer    | Total delivery attempts for `http` and `slack` targets, not retries on top of the first: 0 is a single fire-and-forget attempt with no retries and no circuit breaker, and a value of N makes N attempts in all, with exponential backoff and a per-target circuit breaker. Ignored by `database` and `log` targets |
-| `max_queue_size` | integer    | Stored and shown on the target's detail view, but not enforced anywhere yet: nothing in the delivery engine consults it. Queue depth is set by the two fixed 10,000-entry channels |
 
 **Relations:** Belongs to Webhook. Has many Deliveries.
 
@@ -2370,12 +2369,16 @@ the breaker turns it away.
 While a target's breaker is open, the target's row on the webhook page
 says its deliveries are paused until the cooldown ends, in UTC and as a
 time from now. Each of its `retrying` deliveries shows as waiting in the
-event log and on the event's page, with the time it will be tried next:
+event log and on the event's page, with the earliest it can be tried
-the later of the cooldown's end and the end of its own backoff after its
+next: the later of the cooldown's end and the end of its own backoff
-last attempt. While the breaker is half-open, the row says instead that
+after its last attempt. It is only the earliest: when the cooldown ends,
-deliveries are held while one delivery tests whether the target has
+one of the target's waiting deliveries is sent to test it while the
-recovered, with no time, and the target's deliveries show their plain
+others wait at least one more cooldown, as the row also says. A time not
-status, since any of them may be the one being sent.
+on the current UTC day is shown with its date. While the breaker is
+half-open, the row says instead that deliveries are held while one
+delivery tests whether the target has recovered, with no time, and the
+target's deliveries show their plain status, since any of them may be
+the one being sent.
 
 ### Metrics
 

internal/database/model_target.go

@@ -31,8 +31,7 @@ type Target struct {
 
 	// For HTTP targets (max_retries=0 means fire-and-forget,
 	// >0 enables retries with backoff)
-	MaxRetries   int `json:"maxRetries,omitempty"`
+	MaxRetries int `json:"maxRetries,omitempty"`
-	MaxQueueSize int `json:"maxQueueSize,omitempty"`
 
 	// Relations. No model marshals the record it belongs to:
 	// Webhook.Targets leads back here, and the JSON could loop.

internal/delivery/engine.go

@@ -311,13 +311,11 @@ func (e *Engine) Rename(
 	return e.dbTarget.rename(targetID, webhookName, targetName)
 }
 
-// StateAndCooldown implements CircuitBreakers. It is
+// StateAndCooldown implements CircuitBreakers. It returns the state of
-// CircuitBreaker.StateAndCooldown for the target's breaker. While the
+// the target's circuit breaker and, while the breaker is open, what is
-// breaker is open, the pages show the target's deliveries as paused
+// left of its cooldown; the cooldown is zero once that has passed and
-// until its cooldown ends; while it is half-open, they show them as
+// in any other state. A target with no breaker reads as closed with no
-// held, with no time, while one delivery tests whether the target has
+// cooldown, and reading never creates one.
-// recovered. A target with no breaker reads as closed, and reading
-// never creates one.
 func (e *Engine) StateAndCooldown(
 	targetID string,
 ) (CircuitState, time.Duration) {

internal/delivery/target_config_view.go

@@ -173,13 +173,6 @@ func httpConfigFields(t *database.Target) []ConfigField {
 
 	fields = append(fields, maxRetriesField(t))
 
-	if t.MaxQueueSize > 0 {
-		fields = append(fields, ConfigField{
-			Label: "Max Queue Size",
-			Value: strconv.Itoa(t.MaxQueueSize),
-		})
-	}
-
 	return fields
 }
 

internal/delivery/target_config_view_test.go

@@ -197,14 +197,12 @@ func TestNewTargetViews_Slack(t *testing.T) {
 }
 
 // TestNewTargetViews_SlackRetries proves a Slack target shows
-// its retry count the same way an HTTP target does, and no
+// its retry count the same way an HTTP target does.
-// queue size even when one is stored: delivery never reads it.
 func TestNewTargetViews_SlackRetries(t *testing.T) {
 	t.Parallel()
 
 	target := slackTarget()
 	target.MaxRetries = 2
-	target.MaxQueueSize = 100
 
 	view := viewFor(t, target)
 
@@ -226,8 +224,7 @@ func TestNewTargetViews_HTTP(t *testing.T) {
 		Config: `{"url":"` + viewExampleHook + `",` +
 			`"timeout":30,` +
 			`"headers":{"Authorization":"Bearer sekrit"}}`,
-		MaxRetries:   5,
+		MaxRetries: 5,
-		MaxQueueSize: 100,
 	})
 
 	fields := fieldMap(view.Config)
@@ -239,7 +236,6 @@ func TestNewTargetViews_HTTP(t *testing.T) {
 			"Timeout":         "30s",
 			"Headers":         "1 configured",
 			viewMaxRetries:    "5",
-			"Max Queue Size":  "100",
 		},
 		fields,
 	)

internal/handlers/handlers.go

@@ -112,22 +112,25 @@ type Handlers struct {
 // parsePageTemplate parses a page-specific template set from the
 // embedded FS. Each page template is combined with the shared
 // base, htmlheader, navbar and notice templates, and with any further
-// files the page includes. The page file must be listed first so that
+// files the page includes. The set is named after the page file, so
-// its root action ({{template "base" .}}) becomes the template set's
+// the page's root action ({{template "base" .}}) is its entry point.
-// entry point.
+//
+// The page file is parsed last because a later definition of a name
+// replaces an earlier one: the page's {{define "title"}} must replace
+// the {{block "title"}} fallback in htmlheader.html.
 func parsePageTemplate(
 	pageFile string, included ...string,
 ) *template.Template {
 	files := append([]string{
-		pageFile,
 		"base.html",
 		"htmlheader.html",
 		"navbar.html",
 		"notice.html",
 	}, included...)
+	files = append(files, pageFile)
 
 	return template.Must(
-		template.ParseFS(templates.Templates, files...),
+		template.New(pageFile).ParseFS(templates.Templates, files...),
 	)
 }
 

internal/handlers/page_title_test.go

@@ -0,0 +1,107 @@
+package handlers_test
+
+import (
+	"html/template"
+	"net/http"
+	"strings"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"sneak.berlin/go/webhooker/internal/database"
+	"sneak.berlin/go/webhooker/internal/handlers"
+	"sneak.berlin/go/webhooker/internal/session"
+	"sneak.berlin/go/webhooker/templates"
+)
+
+// TestEveryPageRendersItsOwnTitle renders each page template and checks
+// the browser tab title is the one the page declares, not the
+// "Webhooker" fallback in htmlheader.html. A page that fails to render
+// shows the error page's title instead, and fails here too.
+func TestEveryPageRendersItsOwnTitle(t *testing.T) {
+	t.Parallel()
+
+	var h *handlers.Handlers
+
+	var sess *session.Session
+
+	app := newTestApp(t, &h, &sess)
+	app.RequireStart()
+
+	t.Cleanup(app.RequireStop)
+
+	// A pointer, as in the handlers: some pages call
+	// Webhook.RetentionLabel, a pointer method.
+	webhook := &database.Webhook{Name: "orders", RetentionDays: 14}
+	webhook.ID = testWebhookID
+
+	pages := []struct {
+		page  string
+		data  map[string]any
+		title string
+	}{
+		{"login.html", map[string]any{}, "Login - Webhooker"},
+		{"profile.html", map[string]any{}, "Profile - Webhooker"},
+		{"settings.html", map[string]any{}, "Settings - Webhooker"},
+		{"sources_list.html", map[string]any{}, "Webhooks - Webhooker"},
+		{"sources_new.html", map[string]any{}, "New Webhook - Webhooker"},
+		{
+			"source_detail.html",
+			map[string]any{dataKeyWebhook: webhook},
+			"orders - Webhooker",
+		},
+		{
+			"source_edit.html",
+			map[string]any{dataKeyWebhook: webhook},
+			"Edit orders - Webhooker",
+		},
+		{
+			"source_logs.html",
+			map[string]any{dataKeyWebhook: webhook, "TotalEvents": int64(0)},
+			"Full Event Log - orders - Webhooker",
+		},
+		{
+			"event_detail.html",
+			map[string]any{dataKeyWebhook: webhook},
+			"Event - orders - Webhooker",
+		},
+		{
+			"target_edit.html",
+			map[string]any{
+				dataKeyWebhook: webhook,
+				"Target":       map[string]any{"Name": "alerts", "Type": "slack"},
+			},
+			"Edit alerts - Webhooker",
+		},
+		{
+			"error.html",
+			map[string]any{"StatusText": http.StatusText(http.StatusNotFound)},
+			"Not Found - Webhooker",
+		},
+	}
+
+	for _, p := range pages {
+		body := renderPage(t, h, sess, p.page, p.data)
+
+		_, afterOpen, _ := strings.Cut(body, "<title>")
+		title, _, _ := strings.Cut(afterOpen, "</title>")
+
+		assert.Equal(t, p.title, title, p.page)
+	}
+}
+
+// TestTitleFallbackIsWebhooker checks the title htmlheader.html gives a
+// page that declares none. Every page declares one, so it is checked on
+// htmlheader.html alone.
+func TestTitleFallbackIsWebhooker(t *testing.T) {
+	t.Parallel()
+
+	header := template.Must(
+		template.ParseFS(templates.Templates, "htmlheader.html"),
+	)
+
+	var buf strings.Builder
+
+	require.NoError(t, header.ExecuteTemplate(&buf, "htmlheader", nil))
+	assert.Contains(t, buf.String(), "<title>Webhooker</title>")
+}

internal/handlers/target_list.go

@@ -31,10 +31,12 @@ type TargetRowView struct {
 }
 
 // PausedView is a target's circuit breaker turning deliveries away.
-// While the breaker is open, Until is when they resume, in UTC, and
+// While the breaker is open, Until is a time in UTC, and Relative how
-// Relative how long that is from now. While it is half-open both are
+// long that is from now: on the target's row, when the cooldown ends;
-// empty: the cooldown has ended, and the target's deliveries are held
+// on a delivery, the earliest it can be tried next. While it is
-// while one delivery tests whether the target has recovered.
+// half-open both are empty: the cooldown has ended, and the target's
+// deliveries are held while one delivery tests whether the target has
+// recovered.
 type PausedView struct {
 	Until    string
 	Relative string
@@ -56,11 +58,13 @@ func (h *Handlers) pausedView(targetID string) *PausedView {
 }
 
 // deliveryPausedView reads the circuit breaker of a retrying delivery's
-// target. While it is open, it says when the delivery will be tried
+// target. While it is open, it says the earliest the delivery can be
-// next: the later of the cooldown's end and the end of the delivery's
+// tried next: the later of the cooldown's end and the end of the
-// own backoff after its last attempt. Otherwise it returns nil, half-open
+// delivery's own backoff after its last attempt. It is only the
-// included, since the delivery may then be the one being sent to test
+// earliest: when the cooldown ends, one of the target's waiting
-// the target.
+// deliveries is sent to test it while the others wait at least one more
+// cooldown. Otherwise it returns nil, half-open included, since the
+// delivery may then be the one being sent to test the target.
 func (h *Handlers) deliveryPausedView(
 	targetID string, attempts []deliveryResultRow,
 ) *PausedView {
@@ -83,10 +87,19 @@ func (h *Handlers) deliveryPausedView(
 	return newPausedView(next)
 }
 
-// newPausedView is a PausedView of deliveries that resume at until.
+// newPausedView is a PausedView of deliveries paused until the given
+// time. A time not on the current UTC day is written with its date, as
+// the event log writes its times.
 func newPausedView(until time.Time) *PausedView {
+	until = until.UTC()
+
+	layout := time.TimeOnly
+	if until.Format(time.DateOnly) != time.Now().UTC().Format(time.DateOnly) {
+		layout = time.DateTime
+	}
+
 	return &PausedView{
-		Until:    until.UTC().Format(time.TimeOnly) + " UTC",
+		Until:    until.Format(layout) + " UTC",
 		Relative: humanize.Time(until),
 	}
 }

internal/handlers/target_paused_test.go

@@ -15,22 +15,24 @@ import (
 	"sneak.berlin/go/webhooker/internal/session"
 )
 
-// resumesAt is how the pages write when a paused target's deliveries
+// cooldownEnds is how the pages write the end of a paused target's
-// resume at the end of its breaker's cooldown: the time in UTC, then
+// breaker's cooldown: the time in UTC, with its date when that falls on
-// how long that is from now.
+// another UTC day, then how long that is from now.
-const resumesAt = `\d\d:\d\d:\d\d UTC \(\d+ seconds from now\)`
+const cooldownEnds = `(\d{4}-\d\d-\d\d )?\d\d:\d\d:\d\d UTC ` +
+	`\(\d+ seconds from now\)`
 
 // TestPausedTarget_ShownUntilBreakerCloses takes an http target's
 // circuit breaker from open through half-open to closed.
 //
 // Open, the target's row on the webhook page says its deliveries are
 // paused and until when, and each retrying delivery says it is waiting
-// and why in the event log and on the event's page, until the later of
+// and why in the event log and on the event's page, with the earliest
-// the cooldown's end and the end of its own backoff. Half-open, the row
+// it can be tried next: the later of the cooldown's end and the end of
-// says deliveries are held while one delivery tests the target, with no
+// its own backoff, with the date when that is another UTC day.
-// time, and no delivery says it is waiting. Closed, the pages say
+// Half-open, the row says deliveries are held while one delivery tests
-// neither. The delivered delivery and the log target are shown as
+// the target, with no time, and no delivery says it is waiting. Closed,
-// before throughout.
+// the pages say neither. The delivered delivery and the log target are
+// shown as before throughout.
 func TestPausedTarget_ShownUntilBreakerCloses(t *testing.T) {
 	t.Parallel()
 
@@ -59,38 +61,40 @@ func TestPausedTarget_ShownUntilBreakerCloses(t *testing.T) {
 	addDelivery(t, dbMgr, wh.ID, delivered.ID, target.ID,
 		database.DeliveryStatusDelivered)
 
-	// This delivery's 13th attempt failed a minute ago, so its own
+	// This delivery's 18th attempt failed a minute ago, so its own
-	// backoff ends over an hour from now, long after the cooldown.
+	// backoff ends over a day from now: long after the cooldown, and on
+	// another UTC day, so the page shows the date.
 	backedOff := seedStoredEvent(t, dbMgr, wh.ID, `{"n":3}`)
 	backedOffID := addDelivery(t, dbMgr, wh.ID, backedOff.ID, target.ID,
 		database.DeliveryStatusRetrying)
 
 	failedAt := time.Now().Add(-time.Minute).Truncate(time.Second)
-	addFailedAttempt(t, dbMgr, wh.ID, backedOffID, 13, failedAt)
+	addFailedAttempt(t, dbMgr, wh.ID, backedOffID, 18, failedAt)
 
-	backoffEnds := failedAt.Add(delivery.Backoff(13)).UTC().
+	backoffEnds := failedAt.Add(delivery.Backoff(18)).UTC().
-		Format(time.TimeOnly) + " UTC (1 hour from now)"
+		Format("2006-01-02 15:04:05") + " UTC (1 day from now)"
 
 	const waiting = "waiting: target paused after repeated failures, " +
-		"resumes "
+		"next try no earlier than "
 
 	breakers.Set(target.ID, delivery.CircuitOpen, 30*time.Second)
 
 	list := targetList(t, renderSourceDetailPage(t, h, sess, wh.ID))
 	assert.Regexp(t, "t-http http Active Edit Deactivate Delete "+
-		"Deliveries Paused: after repeated failures, until "+resumesAt,
+		"Deliveries Paused: after repeated failures, until "+cooldownEnds+
-		list)
+		", then one waiting delivery is sent to test the target while "+
+		"the others wait at least one more cooldown", list)
 	assert.Equal(t, 1, strings.Count(list, "Paused"))
 
 	log := renderSourceLogsPage(t, h, sess, wh.ID)
 	assert.Equal(t, 2, strings.Count(log, "t-http: waiting"))
 	assert.Contains(t, log, "t-http: delivered")
-	assert.Regexp(t, waiting+resumesAt, log)
+	assert.Regexp(t, waiting+cooldownEnds, log)
 	assert.Contains(t, log, waiting+backoffEnds)
 	assert.NotContains(t, log, "retrying")
 
 	page := eventPage(t, h, sess, wh.ID, retrying.ID)
-	assert.Regexp(t, waiting+resumesAt, page)
+	assert.Regexp(t, waiting+cooldownEnds, page)
 	assert.NotContains(t, page, "retrying")
 
 	page = eventPage(t, h, sess, wh.ID, backedOff.ID)

templates/event_detail.html

@@ -69,7 +69,7 @@
                 <div class="flex flex-wrap items-center justify-between gap-3">
                     <span class="text-sm text-gray-700">{{.Target.DisplayName}}</span>
                     <span class="flex flex-wrap items-center gap-3">
-                        <span class="text-xs {{if eq .Status "delivered"}}text-green-600{{else if eq .Status "failed"}}text-red-600{{else if eq .Status "retrying"}}text-yellow-600{{else}}text-gray-400{{end}}">{{with .Paused}}waiting: target paused after repeated failures, resumes {{.Until}} ({{.Relative}}){{else}}{{.Status}}{{end}}</span>
+                        <span class="text-xs {{if eq .Status "delivered"}}text-green-600{{else if eq .Status "failed"}}text-red-600{{else if eq .Status "retrying"}}text-yellow-600{{else}}text-gray-400{{end}}">{{with .Paused}}waiting: target paused after repeated failures, next try no earlier than {{.Until}} ({{.Relative}}){{else}}{{.Status}}{{end}}</span>
                         <span class="text-xs text-gray-400">{{.AttemptCount}} attempt{{if ne .AttemptCount 1}}s{{end}}</span>
                     </span>
                 </div>

templates/source_detail.html

@@ -258,7 +258,7 @@
                     {{with .Paused}}
                     <div class="text-xs text-yellow-600 mt-1">
                         <span class="font-medium">Deliveries Paused:</span>
-                        <span>{{if .Until}}after repeated failures, until {{.Until}} ({{.Relative}}){{else}}held while one delivery tests whether the target has recovered{{end}}</span>
+                        <span>{{if .Until}}after repeated failures, until {{.Until}} ({{.Relative}}), then one waiting delivery is sent to test the target while the others wait at least one more cooldown{{else}}held while one delivery tests whether the target has recovered{{end}}</span>
                     </div>
                     {{end}}
                     {{range .Config}}

templates/source_logs.html

@@ -65,7 +65,7 @@
                                     <button type="button" class="btn-small flex-1 flex-wrap justify-between gap-2 text-left" @click="toggle">
                                         <span class="flex flex-wrap items-center gap-3">
                                             <span class="text-sm text-gray-700">{{.Target.DisplayName}}</span>
-                                            <span class="text-xs {{if eq .Status "delivered"}}text-green-600{{else if eq .Status "failed"}}text-red-600{{else if eq .Status "retrying"}}text-yellow-600{{else}}text-gray-400{{end}}">{{with .Paused}}waiting: target paused after repeated failures, resumes {{.Until}} ({{.Relative}}){{else}}{{.Status}}{{end}}</span>
+                                            <span class="text-xs {{if eq .Status "delivered"}}text-green-600{{else if eq .Status "failed"}}text-red-600{{else if eq .Status "retrying"}}text-yellow-600{{else}}text-gray-400{{end}}">{{with .Paused}}waiting: target paused after repeated failures, next try no earlier than {{.Until}} ({{.Relative}}){{else}}{{.Status}}{{end}}</span>
                                         </span>
                                         <span class="flex flex-wrap items-center gap-3">
                                             <span class="text-xs text-gray-400">{{.AttemptCount}} attempt{{if ne .AttemptCount 1}}s{{end}}</span>