Serve /metrics from a registry of its own (closes #227)

The HTTP metrics recorder, the delivery collectors and the Go and
process collectors now register on one prometheus.Registry that fx
provides, instead of Prometheus's global default registry, and
/metrics serves that registry. A second metrics-enabled router in one
process, or the server tests run with -count=2, no longer panics on a
duplicate registration.

The middleware builds its recorder once, in New, so installing
Metrics() on more than one router over the same graph is also safe;
NewForTest gives its Middleware a recorder on a fresh registry. The
scrape keeps the same series and labels, including go_*, process_*
and promhttp_metric_handler_*.

Model: opus-5-5

README.md

@@ -1070,7 +1070,7 @@ unconditionally against whatever files it finds:
 - the main database on connect — `Setting`, `User`, `APIKey`, `Webhook`,
   `Entrypoint`, `Target`
 - each event database when it is lazily opened — `Event`, `Delivery`,
-  `DeliveryResult`, `Totals`
+  `DeliveryResult`
 - each archive database on every open and reopen
 
 There is no schema version table, no migration ledger, and no down
@@ -1384,7 +1384,7 @@ The codebase uses consistent naming throughout (rename completed in
 
 ### Data Model
 
-webhooker's data model has ten entities organized into two tiers: the
+webhooker's data model has nine entities organized into two tiers: the
 **application tier** (user and webhook configuration) and the **event
 tier** (event ingestion, delivery, and logging).
 
@@ -1413,10 +1413,6 @@ tier** (event ingestion, delivery, and logging).
 │  ┌──────────┐       ┌──────────┐       ┌─────────────────┐  │
 │  │  Event   │──1:N──│ Delivery │──1:N──│ DeliveryResult  │  │
 │  └──────────┘       └──────────┘       └─────────────────┘  │
-│                                                             │
-│  ┌──────────┐                                               │
-│  │  Totals  │  (one row of running counts)                  │
-│  └──────────┘                                               │
 └─────────────────────────────────────────────────────────────┘
 ```
 
@@ -1669,7 +1665,6 @@ status across potentially multiple attempts.
 | `event_id` | UUID           | Foreign key → Event |
 | `target_id`| UUID           | Foreign key → Target |
 | `status`   | DeliveryStatus | One of: `pending`, `delivered`, `failed`, `retrying` |
-| `finished_at` | timestamp   | When the delivery became `delivered` or `failed` (nullable; empty while `pending` or `retrying`) |
 
 **Relations:** Belongs to Event. Belongs to Target. Has many
 DeliveryResults.
@@ -1737,29 +1732,6 @@ retries) is individually logged for full observability.
 
 **Relations:** Belongs to Delivery.
 
-#### Totals
-
-The one row of running counts in each event database, read by the
-statistics pane at the top of the webhook page.
-
-| Field                | Type    | Description |
-| -------------------- | ------- | ----------- |
-| `events`             | integer | Events ever stored, resubmitted copies included |
-| `deliveries`         | integer | Deliveries ever created, replays included |
-| `failures`           | integer | Deliveries that ever became `failed` |
-| `events_removed`     | integer | Events retention has deleted |
-| `deliveries_removed` | integer | Deliveries retention has deleted |
-| `failures_removed`   | integer | Failed deliveries retention has deleted |
-
-Each count changes in the transaction that writes or deletes the rows it
-counts. The pane shows each of the first three as a lifetime figure, and
-less what retention removed as the figure within retention, so neither
-needs the rows themselves. Its last-10-minutes and last-24-hours figures
-are counted from the `events` and `deliveries` indexes over just that
-window. Its failure percentage for a window is the deliveries that became
-`failed` in it out of all that became `delivered` or `failed` in it, and
-a dash when none did.
-
 #### Event-tier indexes
 
 These indexes on the per-webhook event databases are declared in the model
@@ -1767,10 +1739,10 @@ tags, so `AutoMigrate` creates them on a fresh and on an existing database:
 
 | Table              | Columns                     | Serves |
 | ------------------ | --------------------------- | ------ |
-| `deliveries`       | `status`, `deleted_at`, `finished_at` | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status, and the webhook page's statistics, which count deliveries by status and when they finished |
+| `deliveries`       | `status`, `deleted_at`      | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status |
 | `deliveries`       | `event_id`, `deleted_at`    | The event log, which loads each event's deliveries, and retention, which selects and deletes the deliveries of expired events |
 | `delivery_results` | `delivery_id`, `deleted_at` | The event log, which loads the attempts of a page's deliveries, and retention, which deletes the attempts of expired events |
-| `events`           | `deleted_at`, `created_at`  | Retention, which selects expired events by age, and the webhook page's statistics, which count recent events and find the newest |
+| `events`           | `deleted_at`, `created_at`  | Retention, which selects expired events by age |
 | `events`           | `created_at`                | Retention's delete of the expired events themselves |
 
 GORM's soft delete adds `deleted_at IS NULL` to these queries; retention's
@@ -1784,10 +1756,9 @@ and SQLite narrows by a `<` only on the last column it uses.
 
 #### Common Fields
 
-Every entity except `Setting` and `Totals` includes these fields from
-`BaseModel`. `Setting` is a bare key-value row with no `id`, no
-timestamps and no soft delete, and `Totals` is a single row of counts
-with only a numeric `id`:
+Every entity except `Setting` includes these fields from `BaseModel`.
+`Setting` is a bare key-value row with no `id`, no timestamps and no
+soft delete:
 
 | Field        | Type      | Description |
 | ------------ | --------- | ----------- |
@@ -1829,7 +1800,6 @@ encryption key is generated and stored, and an `admin` user is created.
 - **Events** — captured incoming webhook payloads
 - **Deliveries** — event-to-target pairings and their status
 - **DeliveryResults** — individual delivery attempt logs
-- **Totals** — running counts of the above, kept through retention
 
 Per-webhook databases are created automatically when a webhook is
 created (and lazily on first access for webhooks that predate this

cmd/webhooker/main.go

@@ -16,6 +16,7 @@ import (
 	"sneak.berlin/go/webhooker/internal/handlers"
 	"sneak.berlin/go/webhooker/internal/healthcheck"
 	"sneak.berlin/go/webhooker/internal/logger"
+	"sneak.berlin/go/webhooker/internal/metrics"
 	"sneak.berlin/go/webhooker/internal/middleware"
 	"sneak.berlin/go/webhooker/internal/resetpw"
 	"sneak.berlin/go/webhooker/internal/server"
@@ -177,6 +178,10 @@ func newApp() *fx.App {
 			healthcheck.New,
 			session.New,
 			handlers.New,
+			// The registry /metrics serves, and the delivery
+			// collectors registered on it.
+			metrics.NewRegistry,
+			metrics.New,
 			middleware.New,
 			// The one SSRF guard both target-creation validation
 			// and the delivery dialer consult, so they cannot

internal/database/event_tier_indexes_test.go

@@ -93,7 +93,6 @@ func TestEventTierQueriesUseTheirIndexes(t *testing.T) {
 		deliveries []database.Delivery
 		results    []database.DeliveryResult
 		depths     []struct{ Depth int }
-		failed     struct{ Count int64 }
 	)
 
 	byStatus := "idx_deliveries_status (status=? AND deleted_at=?)"
@@ -124,7 +123,7 @@ func TestEventTierQueriesUseTheirIndexes(t *testing.T) {
 		Order("attempt_num ASC").Find(&results),
 		"idx_delivery_results_delivery_id (delivery_id=? AND deleted_at=?)")
 
-	// Retention's deletes (deleteExpired), whose subqueries are built
+	// Retention's three deletes (reapExpired), whose subqueries are built
 	// afresh for each statement as it builds them.
 	expiredEventIDs := func() *gorm.DB {
 		return dry.Model(&database.Event{}).Select("id").
@@ -136,12 +135,6 @@ func TestEventTierQueriesUseTheirIndexes(t *testing.T) {
 			Select("id").Where("event_id IN (?)", expiredEventIDs()),
 	).Delete(&database.DeliveryResult{}),
 		"idx_delivery_results_delivery_id (delivery_id=?)", byEvent, byAge)
-	assertPlanUses(t, db, dry.Unscoped().Model(&database.Delivery{}).
-		Select("count(CASE WHEN status = ? THEN 1 END) AS count",
-			database.DeliveryStatusFailed).
-		Where("event_id IN (?)", expiredEventIDs()).
-		Take(&failed),
-		"idx_deliveries_event_id (event_id=?)", byAge)
 	assertPlanUses(t, db, dry.Unscoped().Where(
 		"event_id IN (?)", expiredEventIDs(),
 	).Delete(&database.Delivery{}),
@@ -151,54 +144,6 @@ func TestEventTierQueriesUseTheirIndexes(t *testing.T) {
 	).Delete(&database.Event{}), "idx_events_created_at (created_at<?)")
 }
 
-// TestStatisticsQueriesUseTheirIndexes does the same for the webhook
-// page's statistics (readEventStats in the handlers): deliveries in
-// progress, deliveries finished and events received since a time, and
-// the newest event, which must come straight off an index rather than
-// from sorting every event.
-func TestStatisticsQueriesUseTheirIndexes(t *testing.T) {
-	t.Parallel()
-
-	mgr, lc := setupTestWebhookDBManager(t)
-	ctx := context.Background()
-	require.NoError(t, lc.Start(ctx))
-
-	defer func() { require.NoError(t, lc.Stop(ctx)) }()
-
-	db, err := mgr.GetDB(uuid.New().String())
-	require.NoError(t, err)
-
-	dry := db.Session(&gorm.Session{DryRun: true})
-	since := time.Now()
-
-	var (
-		count  int64
-		newest []time.Time
-	)
-
-	assertPlanUses(t, db, dry.Model(&database.Delivery{}).
-		Where("status IN ?", []database.DeliveryStatus{
-			database.DeliveryStatusPending,
-			database.DeliveryStatusRetrying,
-		}).Count(&count),
-		"idx_deliveries_status (status=? AND deleted_at=?)")
-	assertPlanUses(t, db, dry.Model(&database.Delivery{}).
-		Where("status = ? AND finished_at >= ?",
-			database.DeliveryStatusFailed, since).Count(&count),
-		"idx_deliveries_status "+
-			"(status=? AND deleted_at=? AND finished_at>?)")
-	assertPlanUses(t, db, dry.Model(&database.Event{}).
-		Where("created_at >= ?", since).Count(&count),
-		"idx_events_deleted_at_created_at "+
-			"(deleted_at=? AND created_at>?)")
-
-	newestEvent := dry.Model(&database.Event{}).
-		Order("created_at DESC").Limit(1).Pluck("created_at", &newest)
-	assertPlanUses(t, db, newestEvent,
-		"idx_events_deleted_at_created_at (deleted_at=?)")
-	assert.NotContains(t, queryPlan(t, db, newestEvent), "TEMP B-TREE")
-}
-
 // assertPlanUses asserts that SQLite's plan for a statement GORM built
 // in a dry run, run with the same SQL and arguments GORM would send,
 // names each of the given indexes.
@@ -207,18 +152,6 @@ func assertPlanUses(
 ) {
 	t.Helper()
 
-	plan := queryPlan(t, db, built)
-
-	for _, index := range indexes {
-		assert.Contains(t, plan, index, built.Statement.SQL.String())
-	}
-}
-
-// queryPlan returns SQLite's plan for a statement GORM built in a dry
-// run, run with the same SQL and arguments GORM would send.
-func queryPlan(t *testing.T, db, built *gorm.DB) string {
-	t.Helper()
-
 	var plan []struct{ Detail string }
 
 	require.NoError(t, db.Raw(
@@ -226,5 +159,8 @@ func queryPlan(t *testing.T, db, built *gorm.DB) string {
 		built.Statement.Vars...,
 	).Scan(&plan).Error)
 
-	return fmt.Sprint(plan)
+	for _, index := range indexes {
+		assert.Contains(t, fmt.Sprint(plan), index,
+			built.Statement.SQL.String())
+	}
 }

internal/database/model_delivery.go

@@ -1,10 +1,6 @@
 package database
 
-import (
-	"time"
-
-	"gorm.io/gorm"
-)
+import "gorm.io/gorm"
 
 // DeliveryStatus represents the status of a delivery
 type DeliveryStatus string
@@ -49,12 +45,6 @@ type Delivery struct {
 	// gives.
 	DeletedAt gorm.DeletedAt `gorm:"index:idx_deliveries_event_id,priority:2;index:idx_deliveries_status,priority:2" json:"deletedAt,omitzero"`
 
-	// FinishedAt is when the delivery became delivered or failed, and
-	// nil while it is pending or retrying. It ends the status index,
-	// so the webhook page counts the deliveries that finished in a
-	// recent window by reading that window from the index.
-	FinishedAt *time.Time `gorm:"index:idx_deliveries_status,priority:3" json:"finishedAt,omitempty"`
-
 	// Relations
 	Event           Event            `json:"event,omitzero"`
 	Target          Target           `json:"target,omitzero"`

internal/database/model_totals.go

@@ -1,73 +0,0 @@
-package database
-
-import (
-	"fmt"
-
-	"gorm.io/gorm"
-)
-
-// Totals is the single row of running totals in a webhook's event
-// database. It is what keeps the webhook page's lifetime figures right
-// after retention has removed the rows they count, and what lets the
-// page show them without counting every row.
-//
-// Storing an event, creating a delivery and failing a delivery each
-// add one, and retention adds what it deletes to the Removed columns.
-// Every addition goes through AddTotals, in the transaction that
-// writes or deletes the rows it counts.
-type Totals struct {
-	ID int64 `gorm:"primaryKey"`
-
-	Events     int64 `gorm:"not null"`
-	Deliveries int64 `gorm:"not null"`
-	Failures   int64 `gorm:"not null"`
-
-	EventsRemoved     int64 `gorm:"not null"`
-	DeliveriesRemoved int64 `gorm:"not null"`
-	FailuresRemoved   int64 `gorm:"not null"`
-}
-
-// TableName names the table AddTotals updates.
-func (Totals) TableName() string {
-	return "totals"
-}
-
-// EventsWithinRetention is how many of the webhook's events are still
-// stored.
-func (t Totals) EventsWithinRetention() int64 {
-	return t.Events - t.EventsRemoved
-}
-
-// DeliveriesWithinRetention is how many of the webhook's deliveries
-// are still stored.
-func (t Totals) DeliveriesWithinRetention() int64 {
-	return t.Deliveries - t.DeliveriesRemoved
-}
-
-// FailuresWithinRetention is how many of the webhook's failed
-// deliveries are still stored.
-func (t Totals) FailuresWithinRetention() int64 {
-	return t.Failures - t.FailuresRemoved
-}
-
-// AddTotals adds each count in add to the webhook's running totals.
-// Call it on the transaction that writes or deletes the rows it
-// counts, so the totals change exactly when those rows do.
-func AddTotals(tx *gorm.DB, add Totals) error {
-	err := tx.Exec(
-		`UPDATE totals SET
-			events = events + ?,
-			deliveries = deliveries + ?,
-			failures = failures + ?,
-			events_removed = events_removed + ?,
-			deliveries_removed = deliveries_removed + ?,
-			failures_removed = failures_removed + ?`,
-		add.Events, add.Deliveries, add.Failures,
-		add.EventsRemoved, add.DeliveriesRemoved, add.FailuresRemoved,
-	).Error
-	if err != nil {
-		return fmt.Errorf("adding to running totals: %w", err)
-	}
-
-	return nil
-}

internal/database/models.go

@@ -2,7 +2,7 @@ package database
 
 // Migrate runs database migrations for the main application database.
 // Only configuration-tier models are stored in the main database.
-// Event-tier models (Event, Delivery, DeliveryResult, Totals) live in
+// Event-tier models (Event, Delivery, DeliveryResult) live in
 // per-webhook dedicated databases managed by WebhookDBManager.
 func (d *Database) Migrate() error {
 	return d.db.AutoMigrate(

internal/database/retention.go

@@ -267,101 +267,55 @@ func retentionCutoff(
 
 // reapExpired hard-deletes, in foreign-key-safe order, the delivery
 // results, deliveries, and events associated with events older than
-// cutoff, and adds what it deleted to the running totals, all in one
-// transaction. Deletes are unscoped so rows are physically removed
-// rather than soft-deleted, reclaiming disk. It returns the number of
-// events deleted.
+// cutoff. Deletes are unscoped so rows are physically removed rather
+// than soft-deleted, reclaiming disk. It returns the number of events
+// deleted.
 func reapExpired(db *gorm.DB, cutoff time.Time) (int64, error) {
-	var removed Totals
-
-	err := db.Transaction(func(tx *gorm.DB) error {
-		var err error
-
-		removed, err = deleteExpired(tx, cutoff)
-		if err != nil {
-			return err
-		}
-
-		return AddTotals(tx, removed)
-	})
-	if err != nil {
-		return 0, err
-	}
-
-	return removed.EventsRemoved, nil
-}
-
-// deleteExpired runs reapExpired's deletes and returns how many
-// events, deliveries and failed deliveries they removed.
-func deleteExpired(tx *gorm.DB, cutoff time.Time) (Totals, error) {
-	var removed Totals
-
 	// Fresh subqueries are built per statement to avoid reusing a
 	// mutated builder across executions.
 	expiredEventIDs := func() *gorm.DB {
-		return tx.Model(&Event{}).
+		return db.Model(&Event{}).
 			Select("id").
 			Where("created_at < ?", cutoff)
 	}
 	expiredDeliveryIDs := func() *gorm.DB {
-		return tx.Model(&Delivery{}).
+		return db.Model(&Delivery{}).
 			Select("id").
 			Where("event_id IN (?)", expiredEventIDs())
 	}
 
 	// 1. Delivery results whose delivery belongs to an expired event.
-	res := tx.Unscoped().
+	res := db.Unscoped().
 		Where("delivery_id IN (?)", expiredDeliveryIDs()).
 		Delete(&DeliveryResult{})
 	if res.Error != nil {
-		return removed, fmt.Errorf(
+		return 0, fmt.Errorf(
 			"deleting expired delivery results: %w",
 			res.Error,
 		)
 	}
 
-	// 2. Deliveries belonging to an expired event, after counting the
-	// failed ones among them. The status is tested in the select list
-	// rather than the WHERE clause: there, SQLite would read every
-	// failed delivery the webhook has through the status index,
-	// instead of only the expired ones through the event_id index.
-	var failed struct{ Count int64 }
-
-	err := tx.Unscoped().Model(&Delivery{}).
-		Select("count(CASE WHEN status = ? THEN 1 END) AS count",
-			DeliveryStatusFailed).
-		Where("event_id IN (?)", expiredEventIDs()).
-		Take(&failed).Error
-	if err != nil {
-		return removed, fmt.Errorf(
-			"counting expired failed deliveries: %w", err,
-		)
-	}
-
-	del := tx.Unscoped().
+	// 2. Deliveries belonging to an expired event.
+	del := db.Unscoped().
 		Where("event_id IN (?)", expiredEventIDs()).
 		Delete(&Delivery{})
 	if del.Error != nil {
-		return removed, fmt.Errorf(
+		return 0, fmt.Errorf(
 			"deleting expired deliveries: %w",
 			del.Error,
 		)
 	}
 
 	// 3. The expired events themselves.
-	ev := tx.Unscoped().
+	ev := db.Unscoped().
 		Where("created_at < ?", cutoff).
 		Delete(&Event{})
 	if ev.Error != nil {
-		return removed, fmt.Errorf(
+		return 0, fmt.Errorf(
 			"deleting expired events: %w",
 			ev.Error,
 		)
 	}
 
-	removed.EventsRemoved = ev.RowsAffected
-	removed.DeliveriesRemoved = del.RowsAffected
-	removed.FailuresRemoved = failed.Count
-
-	return removed, nil
+	return ev.RowsAffected, nil
 }

internal/database/totals_test.go

@@ -1,126 +0,0 @@
-package database_test
-
-import (
-	"context"
-	"testing"
-	"time"
-
-	"github.com/google/uuid"
-	"github.com/stretchr/testify/assert"
-	"github.com/stretchr/testify/require"
-	"gorm.io/gorm"
-	"sneak.berlin/go/webhooker/internal/database"
-)
-
-// readTotals reads a webhook database's row of running totals,
-// asserting that it has exactly one.
-func readTotals(t *testing.T, db *gorm.DB) database.Totals {
-	t.Helper()
-
-	var rows []database.Totals
-
-	require.NoError(t, db.Find(&rows).Error)
-	require.Len(t, rows, 1)
-
-	return rows[0]
-}
-
-// TestWebhookDBManager_TotalsRowSurvivesReopen verifies that a new
-// event database starts with one row of zero totals, and that opening
-// it again keeps that row and what was added to it.
-func TestWebhookDBManager_TotalsRowSurvivesReopen(t *testing.T) {
-	t.Parallel()
-
-	mgr, lc := setupTestWebhookDBManager(t)
-	ctx := context.Background()
-	require.NoError(t, lc.Start(ctx))
-
-	defer func() { require.NoError(t, lc.Stop(ctx)) }()
-
-	webhookID := uuid.New().String()
-
-	db, err := mgr.GetDB(webhookID)
-	require.NoError(t, err)
-
-	fresh := readTotals(t, db)
-	assert.Equal(t, database.Totals{ID: fresh.ID}, fresh)
-
-	require.NoError(t, database.AddTotals(db, database.Totals{
-		Events: 2, Deliveries: 3, Failures: 1,
-	}))
-
-	// Drop the cached connection so the next open reopens the file,
-	// as a restart would.
-	require.NoError(t, mgr.CloseAll())
-
-	db, err = mgr.GetDB(webhookID)
-	require.NoError(t, err)
-
-	assert.Equal(t, database.Totals{
-		ID: fresh.ID, Events: 2, Deliveries: 3, Failures: 1,
-	}, readTotals(t, db))
-}
-
-// TestRetentionReaper_AddsWhatItRemovesToTotals verifies that a sweep
-// leaves the lifetime totals alone and adds the events, deliveries and
-// failed deliveries it deletes to the removed totals, so the totals
-// within retention match the rows still stored.
-func TestRetentionReaper_AddsWhatItRemovesToTotals(t *testing.T) {
-	t.Parallel()
-
-	env := setupRetentionTest(t)
-
-	webhookID := createWebhook(t, env.mainDB.DB(), 30)
-
-	db, err := env.mgr.GetDB(webhookID)
-	require.NoError(t, err)
-
-	now := time.Now()
-	expired := now.Add(-40 * 24 * time.Hour)
-
-	seedEventChain(t, db, webhookID, expired)
-	expiredFailure := seedEventChain(t, db, webhookID, expired)
-	recentFailure := seedEventChain(
-		t, db, webhookID, now.Add(-24*time.Hour),
-	)
-
-	for _, id := range []string{
-		expiredFailure.deliveryID, recentFailure.deliveryID,
-	} {
-		require.NoError(t, db.Model(&database.Delivery{}).
-			Where("id = ?", id).
-			Update("status", database.DeliveryStatusFailed).Error)
-	}
-
-	// The totals storing those rows would have left.
-	require.NoError(t, database.AddTotals(db, database.Totals{
-		Events: 3, Deliveries: 3, Failures: 2,
-	}))
-
-	env.reaper.ExportSweep(context.Background())
-
-	totals := readTotals(t, db)
-	assert.Equal(t, database.Totals{
-		ID:     totals.ID,
-		Events: 3, Deliveries: 3, Failures: 2,
-		EventsRemoved: 2, DeliveriesRemoved: 2, FailuresRemoved: 1,
-	}, totals)
-
-	var events, deliveries, failures int64
-
-	require.NoError(t, db.Model(&database.Event{}).Count(&events).Error)
-	require.NoError(t, db.Model(&database.Delivery{}).
-		Count(&deliveries).Error)
-	require.NoError(t, db.Model(&database.Delivery{}).
-		Where("status = ?", database.DeliveryStatusFailed).
-		Count(&failures).Error)
-
-	assert.Equal(t, events, totals.EventsWithinRetention())
-	assert.Equal(t, deliveries, totals.DeliveriesWithinRetention())
-	assert.Equal(t, failures, totals.FailuresWithinRetention())
-
-	// A sweep with nothing left to remove changes nothing.
-	env.reaper.ExportSweep(context.Background())
-
-	assert.Equal(t, totals, readTotals(t, db))
-}

internal/database/webhook_db_manager.go

@@ -35,8 +35,7 @@ var errInvalidCachedDBType = errors.New(
 
 // WebhookDBManager manages per-webhook SQLite database files
 // for event storage. Each webhook gets its own dedicated
-// database containing Events, Deliveries, DeliveryResults and the
-// running Totals of them.
+// database containing Events, Deliveries, and DeliveryResults.
 // Database connections are opened lazily and cached.
 type WebhookDBManager struct {
 	dataDir string
@@ -295,7 +294,7 @@ func (m *WebhookDBManager) openDB(
 
 	// Run migrations for event-tier models only
 	err = db.AutoMigrate(
-		&Event{}, &Delivery{}, &DeliveryResult{}, &Totals{},
+		&Event{}, &Delivery{}, &DeliveryResult{},
 	)
 	if err != nil {
 		_ = sqlDB.Close()
@@ -306,17 +305,6 @@ func (m *WebhookDBManager) openDB(
 		)
 	}
 
-	// A new database gets its row of running totals, all zero.
-	err = db.FirstOrCreate(&Totals{}).Error
-	if err != nil {
-		_ = sqlDB.Close()
-
-		return nil, fmt.Errorf(
-			"creating running totals for webhook database %s: %w",
-			webhookID, err,
-		)
-	}
-
 	m.log.Info(
 		"opened per-webhook database",
 		"webhook_id", webhookID,

internal/delivery/delivery_totals_test.go

@@ -1,100 +0,0 @@
-package delivery_test
-
-import (
-	"testing"
-	"time"
-
-	"github.com/google/uuid"
-	"github.com/stretchr/testify/assert"
-	"github.com/stretchr/testify/require"
-	"gorm.io/gorm"
-	"sneak.berlin/go/webhooker/internal/database"
-)
-
-// failureTotal reads the running failure total of a webhook database.
-func failureTotal(t *testing.T, db *gorm.DB) int64 {
-	t.Helper()
-
-	var totals database.Totals
-
-	require.NoError(t, db.Take(&totals).Error)
-
-	return totals.Failures
-}
-
-// TestUpdateDeliveryStatus_FinishTimeAndFailureTotal pins what a status
-// write records for the webhook page's statistics: the time a delivery
-// finished, set only when it becomes delivered or failed, and one more
-// on the failure total when it fails.
-func TestUpdateDeliveryStatus_FinishTimeAndFailureTotal(t *testing.T) {
-	t.Parallel()
-
-	tests := []struct {
-		status   database.DeliveryStatus
-		finished bool
-		failures int64
-	}{
-		{database.DeliveryStatusRetrying, false, 0},
-		{database.DeliveryStatusDelivered, true, 0},
-		{database.DeliveryStatusFailed, true, 1},
-	}
-
-	for _, tt := range tests {
-		t.Run(string(tt.status), func(t *testing.T) {
-			t.Parallel()
-
-			db := testWebhookDB(t)
-			e := testEngine(t, 1)
-			event := seedEvent(t, db, `{}`)
-			d := seedDelivery(
-				t, db, event.ID, uuid.New().String(),
-				database.DeliveryStatusPending,
-			)
-
-			before := time.Now()
-
-			require.NoError(t, e.ExportUpdateDeliveryStatus(
-				db, &d, tt.status,
-			))
-
-			var stored database.Delivery
-
-			require.NoError(t, db.First(&stored, "id = ?", d.ID).Error)
-			assert.Equal(t, tt.status, stored.Status)
-
-			if tt.finished {
-				require.NotNil(t, stored.FinishedAt)
-				assert.False(t, stored.FinishedAt.Before(before))
-			} else {
-				assert.Nil(t, stored.FinishedAt)
-			}
-
-			assert.Equal(t, tt.failures, failureTotal(t, db))
-		})
-	}
-}
-
-// TestUpdateDeliveryStatus_DeletedDeliveryIsNotCounted covers a
-// delivery retention deleted while the engine still held it. Failing
-// it afterwards writes no row, so it adds no failure either: retention
-// has already counted what it removed.
-func TestUpdateDeliveryStatus_DeletedDeliveryIsNotCounted(t *testing.T) {
-	t.Parallel()
-
-	db := testWebhookDB(t)
-	e := testEngine(t, 1)
-	event := seedEvent(t, db, `{}`)
-	d := seedDelivery(
-		t, db, event.ID, uuid.New().String(),
-		database.DeliveryStatusRetrying,
-	)
-
-	require.NoError(t, db.Unscoped().
-		Delete(&database.Delivery{}, "id = ?", d.ID).Error)
-
-	require.NoError(t, e.ExportUpdateDeliveryStatus(
-		db, &d, database.DeliveryStatusFailed,
-	))
-
-	assert.Zero(t, failureTotal(t, db))
-}

internal/delivery/engine.go

@@ -148,6 +148,7 @@ type EngineParams struct {
 	DBManager *database.WebhookDBManager
 	Logger    *logger.Logger
 	SSRFGuard *Guard
+	Metrics   *metrics.Set
 }
 
 // Engine processes queued deliveries in the background
@@ -167,10 +168,10 @@ type Engine struct {
 	retryCh    chan Task
 	workers    int
 
-	// mtr is the delivery metric set. Production wires the
-	// process-wide one; a test can substitute a set registered on
-	// a private registry so its assertions are not disturbed by
-	// deliveries other tests are making at the same time.
+	// mtr is the delivery metric set. Production wires the one
+	// registered on the registry /metrics serves; a test can
+	// substitute a set registered on a registry it holds, so it can
+	// gather what its own deliveries recorded.
 	mtr *metrics.Set
 
 	// targets maps each target type to its implementation.
@@ -204,7 +205,7 @@ func New(
 		deliveryCh: make(chan Task, deliveryChannelSize),
 		retryCh:    make(chan Task, retryChannelSize),
 		workers:    defaultWorkers,
-		mtr:        metrics.Default(),
+		mtr:        params.Metrics,
 	}
 
 	e.initTargets(&http.Client{
@@ -1554,9 +1555,8 @@ func (e *Engine) updateDeliveryStatus(
 	targetType database.TargetType,
 	status database.DeliveryStatus,
 ) error {
-	err := webhookDB.Transaction(func(tx *gorm.DB) error {
-		return writeDeliveryStatus(tx, d, status)
-	})
+	err := webhookDB.Model(d).
+		Update("status", status).Error
 	if err != nil {
 		return fmt.Errorf(
 			"updating delivery %s to status %s: %w",
@@ -1575,33 +1575,6 @@ func (e *Engine) updateDeliveryStatus(
 	return nil
 }
 
-// writeDeliveryStatus writes a delivery's new status. A delivery that
-// becomes delivered or failed also gets the time it finished, and a
-// failed one is added to the webhook's running failure total. The
-// failure is counted only if the row was still there to update:
-// retention may have deleted it while the engine was working on it.
-func writeDeliveryStatus(
-	tx *gorm.DB,
-	d *database.Delivery,
-	status database.DeliveryStatus,
-) error {
-	columns := map[string]any{"status": status}
-	if status.Terminal() {
-		columns["finished_at"] = time.Now()
-	}
-
-	res := tx.Model(d).Updates(columns)
-	if res.Error != nil {
-		return res.Error
-	}
-
-	if status == database.DeliveryStatusFailed && res.RowsAffected > 0 {
-		return database.AddTotals(tx, database.Totals{Failures: 1})
-	}
-
-	return nil
-}
-
 // settleStatus moves a delivery to its outcome status and reports a
 // failed write through bookkeepingFailed, which leaves the row
 // recoverable. It exists so the target call sites read as one

internal/delivery/engine_test.go

@@ -57,9 +57,7 @@ func testWebhookDB(t *testing.T) *gorm.DB {
 		&database.Event{},
 		&database.Delivery{},
 		&database.DeliveryResult{},
-		&database.Totals{},
 	))
-	require.NoError(t, db.Create(&database.Totals{}).Error)
 
 	return db
 }

internal/delivery/export_test.go

@@ -9,6 +9,7 @@ import (
 	"net/url"
 	"time"
 
+	"github.com/prometheus/client_golang/prometheus"
 	"go.uber.org/fx"
 	"gorm.io/gorm"
 	"sneak.berlin/go/webhooker/internal/database"
@@ -150,16 +151,6 @@ func (e *Engine) ExportDeliverSlack(
 	)
 }
 
-// ExportUpdateDeliveryStatus exposes updateDeliveryStatus. It passes no
-// target type, so no metric moves.
-func (e *Engine) ExportUpdateDeliveryStatus(
-	webhookDB *gorm.DB,
-	d *database.Delivery,
-	status database.DeliveryStatus,
-) error {
-	return e.updateDeliveryStatus(webhookDB, d, "", status)
-}
-
 // ExportProcessNewTask exposes processNewTask.
 func (e *Engine) ExportProcessNewTask(
 	ctx context.Context, task *Task,
@@ -399,7 +390,7 @@ func NewTestEngine(
 		deliveryCh: make(chan Task, deliveryChannelSize),
 		retryCh:    make(chan Task, retryChannelSize),
 		workers:    workers,
-		mtr:        metrics.Default(),
+		mtr:        metrics.New(prometheus.NewRegistry()),
 	}
 	e.initTargets(client)
 
@@ -414,7 +405,7 @@ func NewTestEngineSmallRetry(
 	e := &Engine{
 		log:     log,
 		retryCh: make(chan Task, 1),
-		mtr:     metrics.Default(),
+		mtr:     metrics.New(prometheus.NewRegistry()),
 	}
 	e.initTargets(nil)
 
@@ -437,7 +428,7 @@ func NewTestEngineWithDB(
 		deliveryCh: make(chan Task, deliveryChannelSize),
 		retryCh:    make(chan Task, retryChannelSize),
 		workers:    workers,
-		mtr:        metrics.Default(),
+		mtr:        metrics.New(prometheus.NewRegistry()),
 	}
 	e.initTargets(client)
 
@@ -445,8 +436,7 @@ func NewTestEngineWithDB(
 }
 
 // ExportSetMetrics substitutes the engine's metric set, so a test can
-// assert on collectors registered on a private registry instead of
-// the process-wide ones every other test is also moving.
+// assert on collectors registered on a registry it holds.
 func (e *Engine) ExportSetMetrics(mtr *metrics.Set) {
 	e.mtr = mtr
 }

internal/delivery/metrics_test.go

@@ -35,9 +35,8 @@ const (
 )
 
 // mIsolate gives the setup's engine a metric set registered on a
-// private registry. The process-wide collectors are moved by every
-// other delivery test running in parallel, so exact assertions are
-// only possible against a registry this test owns.
+// registry this test holds, so its exact assertions can gather from
+// it.
 func mIsolate(
 	t *testing.T, s iSetup,
 ) *prometheus.Registry {

internal/handlers/delivery_replay.go

@@ -299,8 +299,7 @@ func countInFlightDeliveries(
 	return count, err
 }
 
-// createReplayDelivery writes the new pending delivery row, adds it to
-// the webhook's running totals in the same transaction, and returns
+// createReplayDelivery writes the new pending delivery row and returns
 // the task that carries it to the delivery engine.
 //
 // The row is written with associations omitted, and neither Event nor
@@ -320,14 +319,7 @@ func createReplayDelivery(
 		Status:   database.DeliveryStatusPending,
 	}
 
-	err := webhookDB.Transaction(func(tx *gorm.DB) error {
-		err := tx.Omit(clause.Associations).Create(dlv).Error
-		if err != nil {
-			return err
-		}
-
-		return database.AddTotals(tx, database.Totals{Deliveries: 1})
-	})
+	err := webhookDB.Omit(clause.Associations).Create(dlv).Error
 	if err != nil {
 		return delivery.Task{}, err
 	}

internal/handlers/export_test.go

@@ -69,21 +69,6 @@ func (s *Handlers) LoadEventLogViewsForTest(
 	return views
 }
 
-// WebhookStatsForTest returns the figures the statistics pane on a
-// webhook's page shows, from the webhook's entrypoints and targets
-// loaded as that page loads them.
-func (s *Handlers) WebhookStatsForTest(webhookID string) *WebhookStats {
-	var entrypoints []database.Entrypoint
-
-	s.db.DB().Where("webhook_id = ?", webhookID).Find(&entrypoints)
-
-	var targets []database.Target
-
-	s.db.DB().Where("webhook_id = ?", webhookID).Find(&targets)
-
-	return s.loadWebhookStats(webhookID, entrypoints, targets)
-}
-
 // AddTemplateForTest registers a template under a page name so that
 // the handlers_test package can drive the render path with a
 // template of its own.

internal/handlers/handlers.go

@@ -12,6 +12,7 @@ import (
 	"net/http"
 	"sync/atomic"
 
+	"github.com/prometheus/client_golang/prometheus"
 	"go.uber.org/fx"
 	"sneak.berlin/go/webhooker/internal/database"
 	"sneak.berlin/go/webhooker/internal/delivery"
@@ -61,6 +62,8 @@ type HandlersParams struct {
 	Notifier     delivery.Notifier
 	Evictor      delivery.WebhookEvictor
 	SSRFGuard    *delivery.Guard
+	Metrics      *metrics.Set
+	Registry     *prometheus.Registry
 }
 
 // Handlers provides HTTP handler methods for all application
@@ -91,22 +94,18 @@ type Handlers struct {
 
 // parsePageTemplate parses a page-specific template set from the
 // embedded FS. Each page template is combined with the shared
-// base, htmlheader, and navbar templates, and with any further files
-// the page includes. The page file must be listed first so that its
-// root action ({{template "base" .}}) becomes the template set's entry
-// point.
-func parsePageTemplate(
-	pageFile string, included ...string,
-) *template.Template {
-	files := append([]string{
-		pageFile,
-		"base.html",
-		"htmlheader.html",
-		"navbar.html",
-	}, included...)
-
+// base, htmlheader, and navbar templates. The page file must be
+// listed first so that its root action ({{template "base" .}})
+// becomes the template set's entry point.
+func parsePageTemplate(pageFile string) *template.Template {
 	return template.Must(
-		template.ParseFS(templates.Templates, files...),
+		template.ParseFS(
+			templates.Templates,
+			pageFile,
+			"base.html",
+			"htmlheader.html",
+			"navbar.html",
+		),
 	)
 }
 
@@ -126,7 +125,7 @@ func New(
 	s.mw = params.Middleware
 	s.notifier = params.Notifier
 	s.evictor = params.Evictor
-	s.mtr = metrics.Default()
+	s.mtr = params.Metrics
 	s.ssrf = params.SSRFGuard
 
 	// Parse all page templates once at startup
@@ -135,7 +134,7 @@ func New(
 		"profile.html":       parsePageTemplate("profile.html"),
 		"sources_list.html":  parsePageTemplate("sources_list.html"),
 		"sources_new.html":   parsePageTemplate("sources_new.html"),
-		"source_detail.html": parsePageTemplate("source_detail.html", "webhook_stats.html"),
+		"source_detail.html": parsePageTemplate("source_detail.html"),
 		"source_edit.html":   parsePageTemplate("source_edit.html"),
 		"source_logs.html":   parsePageTemplate("source_logs.html"),
 		"target_edit.html":   parsePageTemplate("target_edit.html"),

internal/handlers/handlers_test.go

@@ -20,6 +20,7 @@ import (
 	"sneak.berlin/go/webhooker/internal/handlers"
 	"sneak.berlin/go/webhooker/internal/healthcheck"
 	"sneak.berlin/go/webhooker/internal/logger"
+	"sneak.berlin/go/webhooker/internal/metrics"
 	"sneak.berlin/go/webhooker/internal/middleware"
 	"sneak.berlin/go/webhooker/internal/session"
 )
@@ -109,6 +110,8 @@ func newTestApp(
 			func(r *recordingEvictor) delivery.WebhookEvictor {
 				return r
 			},
+			metrics.NewRegistry,
+			metrics.New,
 			middleware.New,
 			delivery.NewGuard,
 			handlers.New,

internal/handlers/metrics.go

@@ -0,0 +1,20 @@
+package handlers
+
+import (
+	"net/http"
+
+	"github.com/prometheus/client_golang/prometheus/promhttp"
+)
+
+// HandleMetrics returns the Prometheus scrape handler for the
+// registry every collector in this process registers on. It is what
+// promhttp.Handler builds for the global default registry, including
+// the promhttp_metric_handler_* series that count scrapes, pointed at
+// that registry instead.
+func (s *Handlers) HandleMetrics() http.HandlerFunc {
+	reg := s.params.Registry
+
+	return promhttp.InstrumentMetricHandler(
+		reg, promhttp.HandlerFor(reg, promhttp.HandlerOpts{}),
+	).ServeHTTP
+}

internal/handlers/source_management.go

@@ -450,7 +450,6 @@ func (h *Handlers) renderSourceDetail(
 		"Targets":     delivery.NewTargetViews(targets),
 		"Events":      events,
 		"BaseURL":     baseURL,
-		"Stats":       h.loadWebhookStats(webhook.ID, entrypoints, targets),
 	}
 
 	h.renderTemplate(w, r, "source_detail.html", data)

internal/handlers/webhook.go

@@ -252,12 +252,11 @@ func requestEventSource(
 	}
 }
 
-// createAndFanOut writes the event and one pending delivery per target,
-// and adds them to the webhook's running totals, in a single
-// transaction, then hands the tasks to the delivery engine. It is the
-// only path by which an event and its deliveries are created, so a
-// resubmitted event is retried, SSRF-guarded and circuit-broken
-// exactly as a received one is.
+// createAndFanOut writes the event and one pending delivery per target
+// in a single transaction, then hands the tasks to the delivery
+// engine. It is the only path by which an event and its deliveries are
+// created, so a resubmitted event is retried, SSRF-guarded and
+// circuit-broken exactly as a received one is.
 //
 // The tasks are returned as well as queued, so a caller can report how
 // many targets the event went to.
@@ -297,16 +296,6 @@ func (h *Handlers) createAndFanOut(
 		return nil, nil, err
 	}
 
-	err = database.AddTotals(tx, database.Totals{
-		Events:     1,
-		Deliveries: int64(len(tasks)),
-	})
-	if err != nil {
-		tx.Rollback()
-
-		return nil, nil, err
-	}
-
 	err = tx.Commit().Error
 	if err != nil {
 		return nil, nil, fmt.Errorf(

internal/handlers/webhook_stats.go

@@ -1,212 +0,0 @@
-package handlers
-
-import (
-	"fmt"
-	"time"
-
-	"gorm.io/gorm"
-	"sneak.berlin/go/webhooker/internal/database"
-)
-
-// The spans of the two recent windows the statistics pane reports on:
-// the last 10 minutes and the last 24 hours.
-const (
-	shortWindow = 10 * time.Minute
-	longWindow  = 24 * time.Hour
-)
-
-// percent turns a fraction into a percentage.
-const percent = 100
-
-// WebhookStats holds the figures in the statistics pane at the top of
-// the webhook page.
-type WebhookStats struct {
-	Entrypoints       int
-	ActiveEntrypoints int
-	Targets           int
-	ActiveTargets     int
-
-	// Totals holds the lifetime counts of events, deliveries and
-	// failures, and how many of each retention has removed.
-	Totals database.Totals
-
-	// InProgress counts the deliveries still pending or retrying.
-	InProgress int64
-
-	// LastEventAt is when the newest stored event arrived, or nil when
-	// none is stored.
-	LastEventAt *time.Time
-
-	Last10Minutes RecentWindow
-	Last24Hours   RecentWindow
-}
-
-// RecentWindow holds what happened in one recent window: the events
-// received in it, and the deliveries that became delivered or failed in
-// it.
-type RecentWindow struct {
-	Events    int64
-	Delivered int64
-	Failed    int64
-}
-
-// FailurePercent is the share of the deliveries finished in the window
-// that failed, or a dash when none finished. Deliveries still pending
-// or retrying are not counted either way.
-func (w RecentWindow) FailurePercent() string {
-	finished := w.Delivered + w.Failed
-	if finished == 0 {
-		return "—"
-	}
-
-	return fmt.Sprintf(
-		"%.1f%%", percent*float64(w.Failed)/float64(finished),
-	)
-}
-
-// loadWebhookStats gathers the figures for the statistics pane from the
-// webhook's entrypoints and targets, as the page has already loaded
-// them, and from its event database. It returns nil, and logs why, when
-// the event database cannot be read.
-func (h *Handlers) loadWebhookStats(
-	webhookID string,
-	entrypoints []database.Entrypoint,
-	targets []database.Target,
-) *WebhookStats {
-	stats := &WebhookStats{
-		Entrypoints: len(entrypoints),
-		Targets:     len(targets),
-	}
-
-	for i := range entrypoints {
-		if entrypoints[i].Active {
-			stats.ActiveEntrypoints++
-		}
-	}
-
-	for i := range targets {
-		if targets[i].Active {
-			stats.ActiveTargets++
-		}
-	}
-
-	// Opening an event database that does not exist would create it,
-	// and it would hold nothing to count.
-	if !h.dbMgr.DBExists(webhookID) {
-		return stats
-	}
-
-	webhookDB, err := h.dbMgr.GetDB(webhookID)
-	if err == nil {
-		err = readEventStats(webhookDB, time.Now(), stats)
-	}
-
-	if err != nil {
-		h.log.Error(
-			"failed to read webhook statistics",
-			"webhook_id", webhookID,
-			"error", err,
-		)
-
-		return nil
-	}
-
-	return stats
-}
-
-// readEventStats fills in the figures that come from the webhook's
-// event database. None of them reads every stored row: the totals are
-// one row, and every other figure is read from an index, over only the
-// rows it counts.
-func readEventStats(
-	db *gorm.DB, now time.Time, stats *WebhookStats,
-) error {
-	err := db.Take(&stats.Totals).Error
-	if err != nil {
-		return fmt.Errorf("reading running totals: %w", err)
-	}
-
-	err = db.Model(&database.Delivery{}).
-		Where("status IN ?", []database.DeliveryStatus{
-			database.DeliveryStatusPending,
-			database.DeliveryStatusRetrying,
-		}).
-		Count(&stats.InProgress).Error
-	if err != nil {
-		return fmt.Errorf("counting deliveries in progress: %w", err)
-	}
-
-	var newest []time.Time
-
-	err = db.Model(&database.Event{}).
-		Order("created_at DESC").
-		Limit(1).
-		Pluck("created_at", &newest).Error
-	if err != nil {
-		return fmt.Errorf("reading newest event time: %w", err)
-	}
-
-	if len(newest) > 0 {
-		stats.LastEventAt = &newest[0]
-	}
-
-	stats.Last10Minutes, err = readRecentWindow(
-		db, now.Add(-shortWindow),
-	)
-	if err != nil {
-		return err
-	}
-
-	stats.Last24Hours, err = readRecentWindow(
-		db, now.Add(-longWindow),
-	)
-
-	return err
-}
-
-// readRecentWindow counts the events received, and the deliveries that
-// became delivered or failed, since the given time.
-func readRecentWindow(
-	db *gorm.DB, since time.Time,
-) (RecentWindow, error) {
-	var w RecentWindow
-
-	err := db.Model(&database.Event{}).
-		Where("created_at >= ?", since).
-		Count(&w.Events).Error
-	if err != nil {
-		return w, fmt.Errorf("counting recent events: %w", err)
-	}
-
-	w.Delivered, err = countFinishedSince(
-		db, database.DeliveryStatusDelivered, since,
-	)
-	if err != nil {
-		return w, err
-	}
-
-	w.Failed, err = countFinishedSince(
-		db, database.DeliveryStatusFailed, since,
-	)
-
-	return w, err
-}
-
-// countFinishedSince counts the deliveries that reached the given
-// final status since the given time.
-func countFinishedSince(
-	db *gorm.DB, status database.DeliveryStatus, since time.Time,
-) (int64, error) {
-	var n int64
-
-	err := db.Model(&database.Delivery{}).
-		Where("status = ? AND finished_at >= ?", status, since).
-		Count(&n).Error
-	if err != nil {
-		return 0, fmt.Errorf(
-			"counting deliveries %s recently: %w", status, err,
-		)
-	}
-
-	return n, nil
-}

internal/handlers/webhook_stats_test.go

@@ -1,328 +0,0 @@
-package handlers_test
-
-import (
-	"net/http"
-	"testing"
-	"time"
-
-	"github.com/google/uuid"
-	"github.com/stretchr/testify/assert"
-	"github.com/stretchr/testify/require"
-	"go.uber.org/fx/fxtest"
-	"gorm.io/gorm"
-	"gorm.io/gorm/clause"
-	"sneak.berlin/go/webhooker/internal/config"
-	"sneak.berlin/go/webhooker/internal/database"
-	"sneak.berlin/go/webhooker/internal/handlers"
-	"sneak.berlin/go/webhooker/internal/logger"
-	"sneak.berlin/go/webhooker/internal/session"
-)
-
-// statsEntrypoint adds an entrypoint to a webhook and returns its path.
-func statsEntrypoint(
-	t *testing.T, db *database.Database, webhookID string, active bool,
-) string {
-	t.Helper()
-
-	ep := &database.Entrypoint{
-		WebhookID: webhookID,
-		Path:      uuid.New().String(),
-	}
-
-	require.NoError(t, db.DB().Omit(clause.Associations).Create(ep).Error)
-	require.NoError(t, db.DB().Model(ep).Update("active", active).Error)
-
-	return ep.Path
-}
-
-// statsDelivery returns the id of an event's delivery to a target.
-func statsDelivery(
-	t *testing.T, webhookDB *gorm.DB, eventID, targetID string,
-) string {
-	t.Helper()
-
-	var d database.Delivery
-
-	require.NoError(t, webhookDB.Where(
-		"event_id = ? AND target_id = ?", eventID, targetID,
-	).First(&d).Error)
-
-	return d.ID
-}
-
-// statsFinish settles a delivery as the delivery engine does: its
-// final status and the time it finished, and for a failure one more on
-// the webhook's failure total, in one transaction.
-func statsFinish(
-	t *testing.T,
-	webhookDB *gorm.DB,
-	deliveryID string,
-	status database.DeliveryStatus,
-	at time.Time,
-) {
-	t.Helper()
-
-	require.NoError(t, webhookDB.Transaction(func(tx *gorm.DB) error {
-		err := tx.Model(&database.Delivery{}).
-			Where("id = ?", deliveryID).
-			Updates(map[string]any{"status": status, "finished_at": at}).
-			Error
-		if err != nil || status != database.DeliveryStatusFailed {
-			return err
-		}
-
-		return database.AddTotals(tx, database.Totals{Failures: 1})
-	}))
-}
-
-// statsAge moves an event's arrival back to the given time.
-func statsAge(
-	t *testing.T, webhookDB *gorm.DB, eventID string, at time.Time,
-) {
-	t.Helper()
-
-	require.NoError(t, webhookDB.Model(&database.Event{}).
-		Where("id = ?", eventID).
-		Update("created_at", at).Error)
-}
-
-// seedStatsHistory builds the webhook the statistics test checks: one
-// day of retention, two entrypoints (one inactive) and three targets
-// (one inactive). Three events arrive through the receiver, and so
-// each has a delivery to the two active targets. The oldest event is
-// past retention, the middle one six hours old, the newest just in.
-// Their deliveries are settled as the delivery engine would, and a
-// replay adds a pending delivery to the oldest event. It returns the
-// webhook, its event database and the newest event.
-func seedStatsHistory(
-	t *testing.T,
-	h *handlers.Handlers,
-	sess *session.Session,
-	db *database.Database,
-	dbMgr *database.WebhookDBManager,
-) (*database.Webhook, *gorm.DB, database.Event) {
-	t.Helper()
-
-	wh := &database.Webhook{
-		UserID: deleteTestUserID, Name: "stats", RetentionDays: 1,
-	}
-	require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
-
-	path := statsEntrypoint(t, db, wh.ID, true)
-	statsEntrypoint(t, db, wh.ID, false)
-
-	first := seedConfiguredTarget(
-		t, db, wh.ID, database.TargetTypeHTTP,
-		`{"url":"`+replayTargetURL+`"}`,
-	)
-	second := seedTarget(t, db, wh.ID, database.TargetTypeLog)
-	inactive := seedTarget(t, db, wh.ID, database.TargetTypeLog)
-	require.NoError(t, db.DB().Model(inactive).
-		Update("active", false).Error)
-
-	router := receiverRouter(h)
-
-	for range 3 {
-		require.Equal(t, http.StatusOK, postReceiver(t, router, path))
-	}
-
-	webhookDB, err := dbMgr.GetDB(wh.ID)
-	require.NoError(t, err)
-
-	events := listEvents(t, webhookDB)
-	require.Len(t, events, 3)
-
-	oldest, middle, newest := events[0], events[1], events[2]
-	now := time.Now()
-
-	statsAge(t, webhookDB, oldest.ID, now.Add(-50*time.Hour))
-	statsAge(t, webhookDB, middle.ID, now.Add(-6*time.Hour))
-
-	oldestFailure := statsDelivery(t, webhookDB, oldest.ID, first.ID)
-	statsFinish(t, webhookDB, oldestFailure,
-		database.DeliveryStatusFailed, now.Add(-49*time.Hour))
-	statsFinish(t, webhookDB,
-		statsDelivery(t, webhookDB, oldest.ID, second.ID),
-		database.DeliveryStatusDelivered, now.Add(-49*time.Hour))
-	statsFinish(t, webhookDB,
-		statsDelivery(t, webhookDB, middle.ID, first.ID),
-		database.DeliveryStatusFailed, now.Add(-5*time.Hour))
-	statsFinish(t, webhookDB,
-		statsDelivery(t, webhookDB, middle.ID, second.ID),
-		database.DeliveryStatusFailed, now.Add(-time.Minute))
-	statsFinish(t, webhookDB,
-		statsDelivery(t, webhookDB, newest.ID, first.ID),
-		database.DeliveryStatusDelivered, now.Add(-2*time.Minute))
-
-	require.Equal(t, http.StatusSeeOther,
-		postReplay(t, h, sess, wh.ID, oldestFailure).Code)
-
-	return wh, webhookDB, newest
-}
-
-// statsPrune runs the real retention reaper until it has removed one
-// event from the webhook's database, then stops it.
-func statsPrune(
-	t *testing.T,
-	db *database.Database,
-	dbMgr *database.WebhookDBManager,
-	log *logger.Logger,
-	webhookDB *gorm.DB,
-) {
-	t.Helper()
-
-	lc := fxtest.NewLifecycle(t)
-	database.NewRetentionReaper(lc, database.RetentionReaperParams{
-		Config: &config.Config{
-			RetentionSweepInterval: 10 * time.Millisecond,
-		},
-		Database:  db,
-		DBManager: dbMgr,
-		Logger:    log,
-	})
-
-	lc.RequireStart()
-
-	require.Eventually(t, func() bool {
-		var totals database.Totals
-
-		err := webhookDB.Take(&totals).Error
-
-		return err == nil && totals.EventsRemoved == 1
-	}, 10*time.Second, 10*time.Millisecond)
-
-	lc.RequireStop()
-}
-
-// assertStatsTotals checks the lifetime events, deliveries and
-// failures, and those within retention.
-func assertStatsTotals(
-	t *testing.T, totals database.Totals, lifetime, within [3]int64,
-) {
-	t.Helper()
-
-	gotLifetime := [3]int64{
-		totals.Events, totals.Deliveries, totals.Failures,
-	}
-	gotWithin := [3]int64{
-		totals.EventsWithinRetention(),
-		totals.DeliveriesWithinRetention(),
-		totals.FailuresWithinRetention(),
-	}
-
-	assert.Equal(t, lifetime, gotLifetime,
-		"lifetime events, deliveries, failures")
-	assert.Equal(t, within, gotWithin,
-		"events, deliveries, failures within retention")
-}
-
-// TestWebhookStats_EveryFigureAcrossRetentionPrune checks every figure
-// the statistics pane shows for the history seedStatsHistory builds,
-// before and after the real retention reaper removes the oldest event.
-func TestWebhookStats_EveryFigureAcrossRetentionPrune(t *testing.T) {
-	t.Parallel()
-
-	var (
-		h     *handlers.Handlers
-		sess  *session.Session
-		db    *database.Database
-		dbMgr *database.WebhookDBManager
-		log   *logger.Logger
-	)
-
-	app := newTestApp(t, &h, &sess, &db, &dbMgr, &log)
-	app.RequireStart()
-
-	t.Cleanup(app.RequireStop)
-
-	wh, webhookDB, newest := seedStatsHistory(t, h, sess, db, dbMgr)
-
-	stats := h.WebhookStatsForTest(wh.ID)
-	require.NotNil(t, stats)
-
-	assert.Equal(t, 2, stats.Entrypoints)
-	assert.Equal(t, 1, stats.ActiveEntrypoints)
-	assert.Equal(t, 3, stats.Targets)
-	assert.Equal(t, 2, stats.ActiveTargets)
-	assertStatsTotals(t, stats.Totals, [3]int64{3, 7, 3}, [3]int64{3, 7, 3})
-	assert.Equal(t, int64(2), stats.InProgress)
-	require.NotNil(t, stats.LastEventAt)
-	assert.True(t, newest.CreatedAt.Equal(*stats.LastEventAt))
-	assert.Equal(t, handlers.RecentWindow{
-		Events: 1, Delivered: 1, Failed: 1,
-	}, stats.Last10Minutes)
-	assert.Equal(t, handlers.RecentWindow{
-		Events: 2, Delivered: 1, Failed: 2,
-	}, stats.Last24Hours)
-	assert.Equal(t, "50.0%", stats.Last10Minutes.FailurePercent())
-	assert.Equal(t, "66.7%", stats.Last24Hours.FailurePercent())
-
-	// Retention removes the oldest event with its three deliveries,
-	// one of them failed and one the pending replay.
-	statsPrune(t, db, dbMgr, log, webhookDB)
-
-	after := h.WebhookStatsForTest(wh.ID)
-	require.NotNil(t, after)
-
-	assertStatsTotals(t, after.Totals, [3]int64{3, 7, 3}, [3]int64{2, 4, 2})
-	assert.Equal(t, int64(1), after.InProgress)
-	assert.Equal(t, stats.LastEventAt, after.LastEventAt)
-	assert.Equal(t, stats.Last10Minutes, after.Last10Minutes)
-	assert.Equal(t, stats.Last24Hours, after.Last24Hours)
-
-	body := renderSourceDetailPage(t, h, sess, wh.ID)
-	assert.Contains(t, body, "Statistics")
-	assert.Contains(t, body, "Within retention")
-	assert.Contains(t, body, "50.0%")
-	assert.Contains(t, body, "66.7%")
-}
-
-// TestWebhookStats_WebhookWithNoEvents covers a webhook whose event
-// database has never been opened: every count is zero, the
-// percentages are a dash, and showing the page does not create the
-// database.
-func TestWebhookStats_WebhookWithNoEvents(t *testing.T) {
-	t.Parallel()
-
-	var (
-		h     *handlers.Handlers
-		sess  *session.Session
-		db    *database.Database
-		dbMgr *database.WebhookDBManager
-	)
-
-	app := newTestApp(t, &h, &sess, &db, &dbMgr)
-	app.RequireStart()
-
-	t.Cleanup(app.RequireStop)
-
-	wh := seedWebhook(t, db)
-
-	assert.Equal(t, &handlers.WebhookStats{}, h.WebhookStatsForTest(wh.ID))
-	assert.Equal(t, "—", handlers.RecentWindow{}.FailurePercent())
-
-	body := renderSourceDetailPage(t, h, sess, wh.ID)
-	assert.Contains(t, body, "Statistics")
-	assert.False(t, dbMgr.DBExists(wh.ID))
-}
-
-// TestRecentWindow_FailurePercent pins the percentage: failed
-// deliveries out of all that finished in the window.
-func TestRecentWindow_FailurePercent(t *testing.T) {
-	t.Parallel()
-
-	tests := []struct {
-		window handlers.RecentWindow
-		want   string
-	}{
-		{handlers.RecentWindow{}, "—"},
-		{handlers.RecentWindow{Events: 4}, "—"},
-		{handlers.RecentWindow{Delivered: 3, Failed: 1}, "25.0%"},
-		{handlers.RecentWindow{Failed: 2}, "100.0%"},
-		{handlers.RecentWindow{Delivered: 2}, "0.0%"},
-	}
-
-	for _, tt := range tests {
-		assert.Equal(t, tt.want, tt.window.FailurePercent(), tt.window)
-	}
-}

internal/metrics/metrics.go

@@ -3,17 +3,17 @@
 // deliveries are attempted, how they end, how long they take, how
 // deep the queues are, and how many circuit breakers are open.
 //
-// The inbound HTTP metrics come from the go-http-metrics recorder in
-// internal/middleware and land on prometheus.DefaultRegisterer. These
-// collectors register there too, so both surfaces are gathered by the
-// one promhttp handler mounted on the authenticated /metrics route.
+// It also builds the registry the authenticated /metrics route
+// serves. These collectors, the inbound HTTP metrics recorded in
+// internal/middleware, and the Go runtime and process collectors all
+// register on that one registry, never on Prometheus's global default.
 package metrics
 
 import (
-	"sync"
 	"time"
 
 	"github.com/prometheus/client_golang/prometheus"
+	"github.com/prometheus/client_golang/prometheus/collectors"
 	"github.com/prometheus/client_golang/prometheus/promauto"
 	"sneak.berlin/go/webhooker/internal/database"
 )
@@ -57,25 +57,31 @@ var knownTargetTypes = []database.TargetType{
 	database.TargetTypeSlack,
 }
 
-// defaultSet is the process-wide metric set, registered on the same
-// registry the HTTP middleware and the /metrics handler already use.
-// It is built on first use rather than in an init so that a test
-// binary that never touches metrics never registers them.
+// NewRegistry returns the registry /metrics serves, carrying the Go
+// runtime and process collectors that Prometheus's global default
+// registry carries, so the go_* and process_* series stay in the
+// scrape.
 //
-//nolint:gochecknoglobals // one process-wide registration, by design
-var defaultSet = sync.OnceValue(func() *Set {
-	return New(prometheus.DefaultRegisterer)
-})
+// A registry of its own, rather than the global default, is what lets
+// two dependency graphs in one process — two tests, say — each
+// register their collectors without the second registration
+// panicking.
+func NewRegistry() *prometheus.Registry {
+	reg := prometheus.NewRegistry()
+	reg.MustRegister(
+		collectors.NewGoCollector(),
+		collectors.NewProcessCollector(
+			collectors.ProcessCollectorOpts{},
+		),
+	)
 
-// Default returns the process-wide metric set.
-func Default() *Set {
-	return defaultSet()
+	return reg
 }
 
 // Set is one registered group of webhooker's delivery collectors.
-// Production uses the single Default set; tests build their own
-// against a private registry so assertions are not disturbed by
-// deliveries other tests are making concurrently.
+// Production builds one on the registry /metrics serves; tests build
+// their own against a private registry so assertions are not
+// disturbed by deliveries other tests are making concurrently.
 type Set struct {
 	eventsReceived      prometheus.Counter
 	deliveryAttempts    *prometheus.CounterVec
@@ -93,7 +99,7 @@ type Set struct {
 // New registers a full set of delivery collectors on reg and returns
 // it. It panics if reg already holds them, which is the intended
 // behaviour for a duplicate registration.
-func New(reg prometheus.Registerer) *Set {
+func New(reg *prometheus.Registry) *Set {
 	factory := promauto.With(reg)
 
 	s := &Set{

internal/middleware/export_test.go

@@ -10,8 +10,7 @@ import (
 
 // MetricsMiddlewareForTest builds the metrics recording middleware
 // against a caller-supplied recorder, so a test can gather from its
-// own Prometheus registry rather than the process-wide default one
-// that Middleware.Metrics uses.
+// own Prometheus registry without building a whole Middleware.
 func MetricsMiddlewareForTest(
 	rec httpmetrics.Recorder,
 ) func(http.Handler) http.Handler {

internal/middleware/metrics.go

@@ -7,7 +7,6 @@ import (
 
 	"github.com/go-chi/chi"
 	httpmetrics "github.com/slok/go-http-metrics/metrics"
-	prommetrics "github.com/slok/go-http-metrics/metrics/prometheus"
 	ghmm "github.com/slok/go-http-metrics/middleware"
 	"github.com/slok/go-http-metrics/middleware/std"
 )
@@ -152,16 +151,14 @@ func (r boundedLabelRecorder) AddInflightRequests(
 var _ httpmetrics.Recorder = boundedLabelRecorder{}
 
 // Metrics returns middleware that records Prometheus HTTP metrics on
-// the default registry, which is the one the /metrics route gathers.
+// the registry the /metrics route serves. Every call shares the one
+// recorder New built, so any number of routers can install it.
 func (s *Middleware) Metrics() func(http.Handler) http.Handler {
-	return metricsMiddleware(
-		prommetrics.NewRecorder(prommetrics.Config{}),
-	)
+	return metricsMiddleware(s.metricsRecorder)
 }
 
 // metricsMiddleware builds the recording middleware against a given
-// recorder, so tests can gather from a registry of their own instead
-// of the process-wide default.
+// recorder, so tests can gather from a registry of their own.
 func metricsMiddleware(
 	rec httpmetrics.Recorder,
 ) func(http.Handler) http.Handler {

internal/middleware/metrics_test.go

@@ -57,9 +57,8 @@ const (
 // Server.setupWebhookRoutes inside it. That ordering is the whole
 // defect, so a test that flattens it would prove nothing.
 //
-// The recorder writes to a registry of the test's own rather than the
-// process-wide default one, so each test observes only its own
-// traffic.
+// The recorder writes to a registry of the test's own, so each test
+// observes only its own traffic.
 func metricsTestRouter(
 	t *testing.T,
 	receiverLimit int,
@@ -455,3 +454,29 @@ func TestMetrics_StatusAndSizeStillRecorded(t *testing.T) {
 		"the interceptor must still count written bytes",
 	)
 }
+
+// TestMetrics_WorksOnNewForTestMiddleware pins that a Middleware built
+// by NewForTest has a recorder of its own: its Metrics() serves a
+// request instead of panicking, and a second one does not collide
+// with the first.
+func TestMetrics_WorksOnNewForTestMiddleware(t *testing.T) {
+	t.Parallel()
+
+	log := slog.New(slog.DiscardHandler)
+	cfg := &config.Config{Environment: "prod"}
+	ok := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = w.Write([]byte(okBody))
+	})
+
+	for range 2 {
+		h := middleware.NewForTest(log, cfg, nil).Metrics()(ok)
+
+		req := httptest.NewRequestWithContext(
+			t.Context(), http.MethodGet, okRoute, nil,
+		)
+		w := httptest.NewRecorder()
+		h.ServeHTTP(w, req)
+
+		assert.Equal(t, http.StatusOK, w.Code)
+	}
+}

internal/middleware/middleware.go

@@ -13,6 +13,9 @@ import (
 	"github.com/go-chi/chi"
 	"github.com/go-chi/chi/middleware"
 	"github.com/go-chi/cors"
+	"github.com/prometheus/client_golang/prometheus"
+	httpmetrics "github.com/slok/go-http-metrics/metrics"
+	prommetrics "github.com/slok/go-http-metrics/metrics/prometheus"
 	"go.uber.org/fx"
 	"sneak.berlin/go/webhooker/internal/config"
 	"sneak.berlin/go/webhooker/internal/globals"
@@ -148,10 +151,11 @@ const (
 type MiddlewareParams struct {
 	fx.In
 
-	Logger  *logger.Logger
-	Globals *globals.Globals
-	Config  *config.Config
-	Session *session.Session
+	Logger   *logger.Logger
+	Globals  *globals.Globals
+	Config   *config.Config
+	Session  *session.Session
+	Registry *prometheus.Registry
 }
 
 // Middleware provides HTTP middleware for logging, CORS, auth, and
@@ -161,6 +165,12 @@ type Middleware struct {
 	params  *MiddlewareParams
 	session *session.Session
 
+	// metricsRecorder records the inbound HTTP metrics on the
+	// registry /metrics serves. It is built once, in New, because
+	// building it registers its collectors, and a second
+	// registration on the same registry panics; see Metrics.
+	metricsRecorder httpmetrics.Recorder
+
 	// loginGuard counts failed credential verifications and bounds
 	// concurrent password hashing. It is built on first use so that
 	// every construction path gets one; see guard().
@@ -179,6 +189,9 @@ func New(
 	s.params = &params
 	s.log = params.Logger.Get()
 	s.session = params.Session
+	s.metricsRecorder = prommetrics.NewRecorder(
+		prommetrics.Config{Registry: params.Registry},
+	)
 
 	return s, nil
 }

internal/middleware/testing.go

@@ -3,12 +3,17 @@ package middleware
 import (
 	"log/slog"
 
+	"github.com/prometheus/client_golang/prometheus"
+	prommetrics "github.com/slok/go-http-metrics/metrics/prometheus"
 	"sneak.berlin/go/webhooker/internal/config"
 	"sneak.berlin/go/webhooker/internal/session"
 )
 
 // NewForTest creates a Middleware with the minimum dependencies
 // needed for testing. This bypasses the fx lifecycle.
+//
+// Its metrics recorder writes to a fresh registry of its own, so
+// Metrics() works on it and two of them never collide.
 func NewForTest(
 	log *slog.Logger,
 	cfg *config.Config,
@@ -20,5 +25,8 @@ func NewForTest(
 			Config: cfg,
 		},
 		session: sess,
+		metricsRecorder: prommetrics.NewRecorder(
+			prommetrics.Config{Registry: prometheus.NewRegistry()},
+		),
 	}
 }

internal/resetpw/resetpw_test.go

@@ -24,6 +24,7 @@ import (
 	"sneak.berlin/go/webhooker/internal/handlers"
 	"sneak.berlin/go/webhooker/internal/healthcheck"
 	"sneak.berlin/go/webhooker/internal/logger"
+	"sneak.berlin/go/webhooker/internal/metrics"
 	"sneak.berlin/go/webhooker/internal/middleware"
 	"sneak.berlin/go/webhooker/internal/resetpw"
 	"sneak.berlin/go/webhooker/internal/session"
@@ -163,6 +164,8 @@ func newServerApp(
 			session.New,
 			func() delivery.Notifier { return &noopNotifier{} },
 			func() delivery.WebhookEvictor { return &noopEvictor{} },
+			metrics.NewRegistry,
+			metrics.New,
 			middleware.New,
 			delivery.NewGuard,
 			handlers.New,

internal/server/routes.go

@@ -7,7 +7,6 @@ import (
 	sentryhttp "github.com/getsentry/sentry-go/http"
 	"github.com/go-chi/chi"
 	"github.com/go-chi/chi/middleware"
-	"github.com/prometheus/client_golang/prometheus/promhttp"
 	"sneak.berlin/go/webhooker/static"
 )
 
@@ -130,12 +129,7 @@ func (s *Server) setupRoutes() {
 	if s.params.Config.MetricsAuthEnabled() {
 		s.router.Group(func(r chi.Router) {
 			r.Use(s.mw.MetricsAuth())
-			r.Get(
-				"/metrics",
-				http.HandlerFunc(
-					promhttp.Handler().ServeHTTP,
-				),
-			)
+			r.Get("/metrics", s.h.HandleMetrics())
 		})
 	}
 

internal/server/routes_test.go

@@ -24,6 +24,7 @@ import (
 	"sneak.berlin/go/webhooker/internal/handlers"
 	"sneak.berlin/go/webhooker/internal/healthcheck"
 	"sneak.berlin/go/webhooker/internal/logger"
+	"sneak.berlin/go/webhooker/internal/metrics"
 	"sneak.berlin/go/webhooker/internal/middleware"
 	"sneak.berlin/go/webhooker/internal/server"
 	"sneak.berlin/go/webhooker/internal/session"
@@ -113,6 +114,8 @@ func newTestEnvWithConfig(
 			session.New,
 			func() delivery.Notifier { return &noopNotifier{} },
 			func() delivery.WebhookEvictor { return &noopEvictor{} },
+			metrics.NewRegistry,
+			metrics.New,
 			middleware.New,
 			delivery.NewGuard,
 			handlers.New,
@@ -1027,3 +1030,46 @@ func TestMetricsRouteUnmountedOnHalfSetConfig(t *testing.T) {
 		})
 	}
 }
+
+// TestTwoMetricsRoutersInOneProcess pins
+// https://git.eeqj.de/sneak/webhooker/issues/227: a second
+// metrics-enabled router in one process used to panic, because the
+// HTTP metrics registered on Prometheus's global default registry.
+// Two routers are built over separate dependency graphs and a third
+// over the first graph again, and each must still serve the HTTP,
+// delivery, Go runtime and process series, and the series counting
+// scrapes of /metrics itself.
+func TestTwoMetricsRoutersInOneProcess(t *testing.T) {
+	t.Parallel()
+
+	first := newTestEnvWithConfig(
+		t, metricsConfig(t, metricsUser, metricsAuthValue),
+	)
+	second := newTestEnvWithConfig(
+		t, metricsConfig(t, metricsUser, metricsAuthValue),
+	)
+	third := &testEnv{
+		router: server.NewRouterForTest(
+			first.log.Get(), first.cfg, first.mw, first.hnd,
+		),
+	}
+
+	for _, env := range []*testEnv{first, second, third} {
+		env.get("/", nil)
+
+		scrape := env.metricsRequest(metricsUser, metricsAuthValue)
+		require.Equal(t, http.StatusOK, scrape.Code)
+
+		for _, series := range []string{
+			"http_request_duration_seconds",
+			"http_response_size_bytes",
+			"http_requests_inflight",
+			"webhooker_events_received_total",
+			"go_goroutines",
+			"process_start_time_seconds",
+			"promhttp_metric_handler_requests_total",
+		} {
+			assert.Contains(t, scrape.Body.String(), series)
+		}
+	}
+}

templates/source_detail.html

@@ -24,8 +24,6 @@
         </div>
     </div>
 
-    {{template "webhook_stats" .}}
-
     <div class="grid grid-cols-1 lg:grid-cols-2 gap-6">
         <!-- Entrypoints -->
         <div class="card">

templates/webhook_stats.html

@@ -1,87 +0,0 @@
-{{define "webhook_stats"}}
-<!-- Statistics pane at the top of the webhook page. -->
-<div class="card mb-6">
-    <div class="p-4 border-b border-gray-200">
-        <h2 class="text-lg font-medium text-gray-900">Statistics</h2>
-    </div>
-    {{with .Stats}}
-    <div class="p-4 flex flex-wrap gap-6 text-sm border-b border-gray-200">
-        <div>
-            <span class="text-gray-500">Entrypoints</span>
-            <span class="font-medium text-gray-900">{{.Entrypoints}}</span>
-            <span class="text-gray-500">({{.ActiveEntrypoints}} active)</span>
-        </div>
-        <div>
-            <span class="text-gray-500">Targets</span>
-            <span class="font-medium text-gray-900">{{.Targets}}</span>
-            <span class="text-gray-500">({{.ActiveTargets}} active)</span>
-        </div>
-        <div>
-            <span class="text-gray-500">Deliveries in progress</span>
-            <span class="font-medium text-gray-900">{{.InProgress}}</span>
-        </div>
-        <div>
-            <span class="text-gray-500">Last event</span>
-            <span class="font-medium text-gray-900">{{with .LastEventAt}}{{.Format "2006-01-02 15:04:05 UTC"}}{{else}}none{{end}}</span>
-        </div>
-        <div>
-            <span class="text-gray-500">Retention</span>
-            <span class="font-medium text-gray-900">{{$.Webhook.RetentionLabel}}</span>
-        </div>
-    </div>
-    <div class="p-4 grid grid-cols-1 lg:grid-cols-2 gap-6 text-sm">
-        <div>
-            <div class="flex py-2 border-b border-gray-200 text-xs text-gray-500 uppercase tracking-wide">
-                <span class="flex-1"></span>
-                <span class="w-32 text-center">Lifetime</span>
-                <span class="w-32 text-center">Within retention</span>
-            </div>
-            <div class="divide-y divide-gray-100">
-                <div class="flex py-2">
-                    <span class="flex-1 text-gray-600">Events</span>
-                    <span class="w-32 text-center text-gray-900">{{.Totals.Events}}</span>
-                    <span class="w-32 text-center text-gray-900">{{.Totals.EventsWithinRetention}}</span>
-                </div>
-                <div class="flex py-2">
-                    <span class="flex-1 text-gray-600">Deliveries</span>
-                    <span class="w-32 text-center text-gray-900">{{.Totals.Deliveries}}</span>
-                    <span class="w-32 text-center text-gray-900">{{.Totals.DeliveriesWithinRetention}}</span>
-                </div>
-                <div class="flex py-2">
-                    <span class="flex-1 text-gray-600">Failures</span>
-                    <span class="w-32 text-center text-gray-900">{{.Totals.Failures}}</span>
-                    <span class="w-32 text-center text-gray-900">{{.Totals.FailuresWithinRetention}}</span>
-                </div>
-            </div>
-        </div>
-        <div>
-            <div class="flex py-2 border-b border-gray-200 text-xs text-gray-500 uppercase tracking-wide">
-                <span class="flex-1"></span>
-                <span class="w-32 text-center">Last 10 minutes</span>
-                <span class="w-32 text-center">Last 24 hours</span>
-            </div>
-            <div class="divide-y divide-gray-100">
-                <div class="flex py-2">
-                    <span class="flex-1 text-gray-600">Events</span>
-                    <span class="w-32 text-center text-gray-900">{{.Last10Minutes.Events}}</span>
-                    <span class="w-32 text-center text-gray-900">{{.Last24Hours.Events}}</span>
-                </div>
-                <div class="flex py-2">
-                    <span class="flex-1 text-gray-600">Failures</span>
-                    <span class="w-32 text-center text-gray-900">{{.Last10Minutes.Failed}}</span>
-                    <span class="w-32 text-center text-gray-900">{{.Last24Hours.Failed}}</span>
-                </div>
-                <div class="flex py-2">
-                    <span class="flex-1 text-gray-600">Failure percentage</span>
-                    <span class="w-32 text-center text-gray-900">{{.Last10Minutes.FailurePercent}}</span>
-                    <span class="w-32 text-center text-gray-900">{{.Last24Hours.FailurePercent}}</span>
-                </div>
-            </div>
-            <p class="mt-2 text-xs text-gray-500">Failure percentage is the failed deliveries out of all deliveries that finished in the window. Deliveries still pending or retrying are not counted.</p>
-        </div>
-    </div>
-    {{else}}
-    <div class="p-4 text-sm text-gray-500">The statistics could not be read.</div>
-    {{end}}
-</div>
-{{end}}