Show 50 recent events with size, processing time and status (closes #347)

The webhook page's list, now headed "50 Most Recent Events", shows the 50 newest events, limited in the query. Each row adds the time received (relative, UTC on hover), the body size, the processing time (slowest delivery, queued to final outcome, retries included, "in progress" while pending), and, when the webhook has exactly one HTTP target, that target's latest HTTP status, coloured by class.

Each event now records its body size in a new body_bytes column when it is stored, so the list never reads bodies; the list loads only the delivery and attempt columns it shows, and a failed load is an error, not an empty list.

An existing event database must be recreated: the column is added in place, with no migration.

Model: opus-5-5

README.md

@@ -145,11 +145,6 @@ TTY detection, and security headers are always applied.
 | `TRUSTED_PROXIES`       | CIDRs whose forwarded headers are trusted (unset: all clients behind a proxy share one rate-limit bucket; a correct login password is never throttled either way) | `""` (none) |
 | `ALLOWED_EGRESS_CIDRS`  | CIDRs that delivery targets may reach despite the SSRF blocklist. Read [Allowing egress to your own network](#allowing-egress-to-your-own-network) before setting it | `""` (none) |
 
-The Settings page of the web UI (`/settings`, behind the login) lists
-every one of these with the value the running server loaded. It is
-read-only, and it shows `METRICS_PASSWORD` and `SENTRY_DSN` only as
-set or not set, never their values.
-
 #### Allowing egress to your own network
 
 By default every delivery target must resolve to a public address. The
@@ -1647,6 +1642,7 @@ data for auditing, for replay, and for resubmission.
 | `headers`      | JSON   | Complete request headers |
 | `body`         | text   | Raw request body |
 | `content_type` | string | Content-Type header value |
+| `body_bytes`   | integer | The body's size in bytes, recorded when the event is stored, on receipt and on resubmit |
 | `resubmitted_from_id` | UUID | The event this one was copied from by a resubmit (nullable; empty for an event that arrived on the receiver). Not a foreign key: the source event can be reaped by retention while its copies remain |
 
 **Relations:** Belongs to Webhook. Belongs to Entrypoint. Has many
@@ -2707,7 +2703,6 @@ abuse limit later; they are tracked as future work.
 | ------ | ------------------------ | ----------- |
 | `GET`  | `/user/{username}`       | User profile page |
 | `POST` | `/user/{username}/password` | Change the user's password (5 per minute per bucket, then `429`; `503` if no verification slot frees up within 5s, or immediately if 16 requests are already queued for one) |
-| `GET`  | `/settings`              | Read-only list of the configuration the server is running with; `METRICS_PASSWORD` and `SENTRY_DSN` show only as set or not set |
 | `GET`  | `/sources`               | List user's webhooks |
 | `GET`  | `/sources/new`           | Create webhook form |
 | `POST` | `/sources/new`           | Create webhook submission |
@@ -2820,7 +2815,6 @@ webhooker/
 │   │   ├── healthcheck.go     # Health check handler
 │   │   ├── index.go           # Index page handler
 │   │   ├── profile.go         # User profile handler
-│   │   ├── settings.go        # Read-only Settings page handler
 │   │   ├── source_management.go  # Webhook CRUD handlers
 │   │   └── webhook.go         # Webhook receiver handler
 │   ├── healthcheck/

go.mod

@@ -4,6 +4,7 @@ go 1.26.1
 
 require (
 	github.com/99designs/basicauth-go v0.0.0-20230316000542-bf6f9cbbf0f8
+	github.com/dustin/go-humanize v1.0.1
 	github.com/getsentry/sentry-go v0.25.0
 	github.com/go-chi/chi v1.5.5
 	github.com/go-chi/cors v1.2.1
@@ -29,7 +30,6 @@ require (
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/cespare/xxhash/v2 v2.2.0 // indirect
 	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
-	github.com/dustin/go-humanize v1.0.1 // indirect
 	github.com/gorilla/securecookie v1.1.2 // indirect
 	github.com/jinzhu/inflection v1.0.0 // indirect
 	github.com/jinzhu/now v1.1.5 // indirect

internal/database/model_event.go

@@ -31,6 +31,11 @@ type Event struct {
 	Body        string `gorm:"type:text"   json:"body"`
 	ContentType string `json:"contentType"`
 
+	// BodyBytes is the size of Body in bytes, recorded when the event
+	// is stored so the recent events list can show it without reading
+	// the body.
+	BodyBytes int64 `gorm:"not null" json:"bodyBytes"`
+
 	// ResubmittedFromID names the event this one was copied from by
 	// an operator resubmit. It is nil for an event that arrived on
 	// the receiver, which is every event created before the column

internal/handlers/event_resubmit_test.go

@@ -204,6 +204,7 @@ func assertEventCopy(
 	assert.Equal(t, original.Method, fresh.Method)
 	assert.Equal(t, original.Headers, fresh.Headers)
 	assert.Equal(t, original.Body, fresh.Body)
+	assert.Equal(t, int64(len(original.Body)), fresh.BodyBytes)
 	assert.Equal(t, original.ContentType, fresh.ContentType)
 	assert.Equal(t, original.EntrypointID, fresh.EntrypointID)
 	assert.Equal(t, original.WebhookID, fresh.WebhookID)

internal/handlers/handlers.go

@@ -13,7 +13,6 @@ import (
 	"sync/atomic"
 
 	"go.uber.org/fx"
-	"sneak.berlin/go/webhooker/internal/config"
 	"sneak.berlin/go/webhooker/internal/database"
 	"sneak.berlin/go/webhooker/internal/delivery"
 	"sneak.berlin/go/webhooker/internal/globals"
@@ -29,7 +28,7 @@ const (
 	// maxBodyShift is the bit shift for 1 MB body limit.
 	maxBodyShift = 20
 	// recentEventLimit is the number of recent events to show.
-	recentEventLimit = 20
+	recentEventLimit = 50
 	// paginationPerPage is the number of items per page.
 	paginationPerPage = 25
 
@@ -54,7 +53,6 @@ type HandlersParams struct {
 
 	Logger       *logger.Logger
 	Globals      *globals.Globals
-	Config       *config.Config
 	Database     *database.Database
 	WebhookDBMgr *database.WebhookDBManager
 	Healthcheck  *healthcheck.Healthcheck
@@ -131,7 +129,6 @@ func New(
 	s.templates = map[string]*template.Template{
 		"login.html":         parsePageTemplate("login.html"),
 		"profile.html":       parsePageTemplate("profile.html"),
-		"settings.html":      parsePageTemplate("settings.html"),
 		"sources_list.html":  parsePageTemplate("sources_list.html"),
 		"sources_new.html":   parsePageTemplate("sources_new.html"),
 		"source_detail.html": parsePageTemplate("source_detail.html"),

internal/handlers/handlers_test.go

@@ -83,25 +83,16 @@ func newTestApp(
 ) *fxtest.App {
 	t.Helper()
 
-	return newTestAppWithConfig(
-		t, &config.Config{DataDir: t.TempDir()}, targets...,
-	)
-}
-
-// newTestAppWithConfig is newTestApp over a caller-supplied Config.
-func newTestAppWithConfig(
-	t *testing.T,
-	cfg *config.Config,
-	targets ...any,
-) *fxtest.App {
-	t.Helper()
-
 	return fxtest.New(
 		t,
 		fx.Provide(
 			globals.New,
 			logger.New,
-			func() *config.Config { return cfg },
+			func() *config.Config {
+				return &config.Config{
+					DataDir: t.TempDir(),
+				}
+			},
 			database.New,
 			database.NewWebhookDBManager,
 			healthcheck.New,

internal/handlers/profile_test.go

@@ -88,6 +88,8 @@ func TestHandleProfile_OwnProfile_OK(t *testing.T) {
 	h.HandleProfile().ServeHTTP(w, req)
 
 	assert.Equal(t, http.StatusOK, w.Code)
+	assert.Contains(t, w.Body.String(), "Account Information")
+	assert.NotContains(t, w.Body.String(), "Account Type")
 }
 
 func TestHandleProfile_OtherProfile_Forbidden(t *testing.T) {

internal/handlers/recent_events.go

@@ -0,0 +1,293 @@
+package handlers
+
+import (
+	"net/http"
+	"slices"
+	"strconv"
+	"time"
+
+	"github.com/dustin/go-humanize"
+	"gorm.io/gorm"
+	"sneak.berlin/go/webhooker/internal/database"
+)
+
+// recentEventColumns is the recent events list's projection. It
+// leaves out the body, for the reason maxRenderedBodyBytes gives,
+// and reads its size from body_bytes, recorded when the event was
+// stored.
+const recentEventColumns = "id, created_at, method, content_type, " +
+	"resubmitted_from_id, body_bytes"
+
+// recentAttemptColumns is the part of a recorded attempt the list
+// uses. The event log's deliveryResultColumns also reads response
+// bodies, which the list does not show.
+const recentAttemptColumns = "delivery_id, status_code, created_at"
+
+// RecentEventView is one row of the recent events list on a
+// webhook's page.
+type RecentEventView struct {
+	Method      string
+	ContentType string
+
+	// ResubmittedFromID names the event this one was copied from,
+	// empty for an event that arrived on the receiver.
+	ResubmittedFromID string
+
+	// Received is how long ago the event arrived, and ReceivedUTC
+	// the full timestamp the page shows on hover.
+	Received    string
+	ReceivedUTC string
+
+	// Size is the size of the stored body.
+	Size string
+
+	// ProcessingTime is how long the event's slowest delivery
+	// took; see processingTime.
+	ProcessingTime string
+
+	// Status is what the webhook's HTTP target answered, and
+	// StatusClass its colour; see targetStatus. Both are empty
+	// unless the webhook has exactly one HTTP target.
+	Status      string
+	StatusClass string
+}
+
+// recentEventRow is one row of recentEventColumns.
+type recentEventRow struct {
+	ID                string
+	CreatedAt         time.Time
+	Method            string
+	ContentType       string
+	ResubmittedFromID *string
+	BodyBytes         uint64
+}
+
+// recentAttemptRow is one row of recentAttemptColumns. CreatedAt is
+// when the attempt's result was recorded, which is when the attempt
+// finished.
+type recentAttemptRow struct {
+	DeliveryID string
+	StatusCode int
+	CreatedAt  time.Time
+}
+
+// singleHTTPTargetID returns the ID of the webhook's HTTP target
+// when it has exactly one, and "" when it has none or several.
+func singleHTTPTargetID(targets []database.Target) string {
+	id := ""
+	count := 0
+
+	for i := range targets {
+		if targets[i].Type == database.TargetTypeHTTP {
+			id = targets[i].ID
+			count++
+		}
+	}
+
+	if count != 1 {
+		return ""
+	}
+
+	return id
+}
+
+// loadRecentEvents loads the webhook's recentEventLimit newest
+// events for its page, newest first. statusTargetID is the
+// webhook's only HTTP target, or "" when the list shows no status.
+func loadRecentEvents(
+	webhookDB *gorm.DB, webhookID, statusTargetID string,
+) ([]RecentEventView, error) {
+	var rows []recentEventRow
+
+	err := webhookDB.Model(&database.Event{}).
+		Select(recentEventColumns).
+		Where("webhook_id = ?", webhookID).
+		Order("created_at DESC").
+		Limit(recentEventLimit).
+		Find(&rows).Error
+	if err != nil {
+		return nil, err
+	}
+
+	eventIDs := make([]string, len(rows))
+	for i := range rows {
+		eventIDs[i] = rows[i].ID
+	}
+
+	// Oldest first, so an event's last delivery to a target is its
+	// newest: a replay adds a delivery rather than changing the
+	// earlier one.
+	var deliveries []database.Delivery
+
+	err = webhookDB.
+		Select("id, event_id, target_id, status, created_at").
+		Where("event_id IN ?", eventIDs).
+		Order("created_at ASC").
+		Find(&deliveries).Error
+	if err != nil {
+		return nil, err
+	}
+
+	byEvent := make(map[string][]database.Delivery, len(rows))
+	deliveryIDs := make([]string, len(deliveries))
+
+	for i := range deliveries {
+		eventID := deliveries[i].EventID
+		byEvent[eventID] = append(byEvent[eventID], deliveries[i])
+		deliveryIDs[i] = deliveries[i].ID
+	}
+
+	attempts, err := loadRecentAttempts(webhookDB, deliveryIDs)
+	if err != nil {
+		return nil, err
+	}
+
+	views := make([]RecentEventView, len(rows))
+	for i := range rows {
+		views[i] = rows[i].view(
+			byEvent[rows[i].ID], attempts, statusTargetID,
+		)
+	}
+
+	return views, nil
+}
+
+// loadRecentAttempts loads the recorded attempts of the listed
+// events' deliveries, keyed by delivery ID, each delivery's in
+// attempt order. The IDs go in chunks for the reason
+// deliveryIDChunkSize gives.
+func loadRecentAttempts(
+	webhookDB *gorm.DB, deliveryIDs []string,
+) (map[string][]recentAttemptRow, error) {
+	byDelivery := make(map[string][]recentAttemptRow)
+
+	for chunk := range slices.Chunk(deliveryIDs, deliveryIDChunkSize) {
+		var rows []recentAttemptRow
+
+		err := webhookDB.Model(&database.DeliveryResult{}).
+			Select(recentAttemptColumns).
+			Where("delivery_id IN ?", chunk).
+			Order("attempt_num ASC").
+			Find(&rows).Error
+		if err != nil {
+			return nil, err
+		}
+
+		for i := range rows {
+			id := rows[i].DeliveryID
+			byDelivery[id] = append(byDelivery[id], rows[i])
+		}
+	}
+
+	return byDelivery, nil
+}
+
+// view projects a loaded row for rendering. deliveries is the
+// event's deliveries, oldest first, and attempts their recorded
+// attempts keyed by delivery ID.
+func (r *recentEventRow) view(
+	deliveries []database.Delivery,
+	attempts map[string][]recentAttemptRow,
+	statusTargetID string,
+) RecentEventView {
+	v := RecentEventView{
+		Method:         r.Method,
+		ContentType:    r.ContentType,
+		Received:       humanize.Time(r.CreatedAt),
+		ReceivedUTC:    r.CreatedAt.UTC().Format(time.DateTime) + " UTC",
+		Size:           humanize.Bytes(r.BodyBytes),
+		ProcessingTime: processingTime(deliveries, attempts),
+	}
+
+	if r.ResubmittedFromID != nil {
+		v.ResubmittedFromID = *r.ResubmittedFromID
+	}
+
+	if statusTargetID != "" {
+		v.Status, v.StatusClass = targetStatus(
+			deliveries, attempts, statusTargetID,
+		)
+	}
+
+	return v
+}
+
+// processingTime is how long the event's slowest delivery took,
+// from being queued to its last recorded attempt, time spent
+// waiting between retries included. A delivery is queued when its
+// event is received, or when an operator replays it, so a replay
+// is timed from the replay rather than from the event's arrival.
+// It is "in progress" while any delivery is pending or retrying,
+// and empty for an event with no deliveries.
+func processingTime(
+	deliveries []database.Delivery,
+	attempts map[string][]recentAttemptRow,
+) string {
+	if len(deliveries) == 0 {
+		return ""
+	}
+
+	var slowest time.Duration
+
+	for i := range deliveries {
+		if !deliveries[i].Status.Terminal() {
+			return "in progress"
+		}
+
+		tries := attempts[deliveries[i].ID]
+		if len(tries) == 0 {
+			continue
+		}
+
+		last := tries[len(tries)-1].CreatedAt
+		slowest = max(slowest, last.Sub(deliveries[i].CreatedAt))
+	}
+
+	return slowest.Round(time.Millisecond).String()
+}
+
+// targetStatus is what the target answered for the event, and the
+// colour to show it in: the HTTP status code of the last attempt of
+// the event's newest delivery to the target. Without a code it is
+// "no response" when that attempt failed before a response
+// arrived, the delivery's status ("pending") before any attempt,
+// and "not sent" when the event has no delivery to the target.
+func targetStatus(
+	deliveries []database.Delivery,
+	attempts map[string][]recentAttemptRow,
+	targetID string,
+) (string, string) {
+	newest := -1
+
+	for i := range deliveries {
+		if deliveries[i].TargetID == targetID {
+			newest = i
+		}
+	}
+
+	if newest < 0 {
+		return "not sent", "text-gray-400"
+	}
+
+	tries := attempts[deliveries[newest].ID]
+	if len(tries) == 0 {
+		return string(deliveries[newest].Status), "text-gray-400"
+	}
+
+	code := tries[len(tries)-1].StatusCode
+
+	switch {
+	case code == 0:
+		return "no response", "text-red-600"
+	case code >= http.StatusInternalServerError:
+		return strconv.Itoa(code), "text-red-600"
+	case code >= http.StatusBadRequest:
+		return strconv.Itoa(code), "text-yellow-600"
+	case code >= http.StatusMultipleChoices:
+		return strconv.Itoa(code), "text-gray-500"
+	case code >= http.StatusOK:
+		return strconv.Itoa(code), "text-green-600"
+	default:
+		return strconv.Itoa(code), "text-gray-500"
+	}
+}

internal/handlers/recent_events_test.go

@@ -0,0 +1,362 @@
+package handlers_test
+
+import (
+	"context"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/go-chi/chi"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"gorm.io/gorm"
+	"gorm.io/gorm/clause"
+	"sneak.berlin/go/webhooker/internal/database"
+	"sneak.berlin/go/webhooker/internal/handlers"
+	"sneak.berlin/go/webhooker/internal/session"
+)
+
+// statusTitle marks the status column's cell in a recent events
+// row; it is absent from the page when the column is not shown.
+const statusTitle = `title="HTTP status from the HTTP target"`
+
+// recentEventsFixture is one started app and a webhook whose
+// recent events list a test fills.
+type recentEventsFixture struct {
+	h         *handlers.Handlers
+	sess      *session.Session
+	db        *database.Database
+	webhook   *database.Webhook
+	webhookDB *gorm.DB
+}
+
+func newRecentEventsFixture(t *testing.T) *recentEventsFixture {
+	t.Helper()
+
+	f := &recentEventsFixture{}
+
+	var dbMgr *database.WebhookDBManager
+
+	app := newTestApp(t, &f.h, &f.sess, &f.db, &dbMgr)
+	app.RequireStart()
+
+	t.Cleanup(app.RequireStop)
+
+	f.webhook = seedWebhook(t, f.db)
+
+	webhookDB, err := dbMgr.GetDB(f.webhook.ID)
+	require.NoError(t, err)
+
+	f.webhookDB = webhookDB
+
+	return f
+}
+
+func (f *recentEventsFixture) render(t *testing.T) string {
+	t.Helper()
+
+	return renderSourceDetailPage(t, f.h, f.sess, f.webhook.ID)
+}
+
+// event records an event received at receivedAt, with its body's
+// size as the receiver records it.
+func (f *recentEventsFixture) event(
+	t *testing.T, contentType, body string, receivedAt time.Time,
+) *database.Event {
+	t.Helper()
+
+	event := &database.Event{
+		WebhookID:   f.webhook.ID,
+		Method:      http.MethodPost,
+		Body:        body,
+		BodyBytes:   int64(len(body)),
+		ContentType: contentType,
+	}
+	event.CreatedAt = receivedAt
+
+	require.NoError(t, f.webhookDB.Omit(
+		clause.Associations,
+	).Create(event).Error)
+
+	return event
+}
+
+// delivery records a delivery of the event to the target, queued
+// when the event was received.
+func (f *recentEventsFixture) delivery(
+	t *testing.T,
+	event *database.Event,
+	targetID string,
+	status database.DeliveryStatus,
+) *database.Delivery {
+	t.Helper()
+
+	return f.deliveryQueuedAt(
+		t, event, targetID, status, event.CreatedAt,
+	)
+}
+
+// deliveryQueuedAt records a delivery of the event to the target,
+// queued at queuedAt, as a replay is.
+func (f *recentEventsFixture) deliveryQueuedAt(
+	t *testing.T,
+	event *database.Event,
+	targetID string,
+	status database.DeliveryStatus,
+	queuedAt time.Time,
+) *database.Delivery {
+	t.Helper()
+
+	dlv := &database.Delivery{
+		EventID:  event.ID,
+		TargetID: targetID,
+		Status:   status,
+	}
+	dlv.CreatedAt = queuedAt
+
+	require.NoError(t, f.webhookDB.Omit(
+		clause.Associations,
+	).Create(dlv).Error)
+
+	return dlv
+}
+
+// attempt records one attempt of the delivery that finished took
+// after the delivery was queued, with HTTP status code (0 for no
+// response).
+func (f *recentEventsFixture) attempt(
+	t *testing.T, dlv *database.Delivery, code int, took time.Duration,
+) {
+	t.Helper()
+
+	result := &database.DeliveryResult{
+		DeliveryID: dlv.ID,
+		AttemptNum: 1,
+		StatusCode: code,
+	}
+	result.CreatedAt = dlv.CreatedAt.Add(took)
+
+	require.NoError(t, f.webhookDB.Omit(
+		clause.Associations,
+	).Create(result).Error)
+}
+
+// statusCell is the status column's cell as the page renders it.
+func statusCell(class, text string) string {
+	return `<span class="font-medium ` + class + `" ` + statusTitle +
+		`>` + text + `</span>`
+}
+
+// TestHandleSourceDetail_ShowsFiftyNewestEvents proves the list is
+// headed "50 Most Recent Events" and holds the 50 newest events,
+// newest first, and not one more.
+func TestHandleSourceDetail_ShowsFiftyNewestEvents(t *testing.T) {
+	t.Parallel()
+
+	f := newRecentEventsFixture(t)
+	base := time.Now().Add(-time.Hour)
+
+	for i := range 51 {
+		f.event(
+			t, fmt.Sprintf("application/x-recent-%02d", i), "{}",
+			base.Add(time.Duration(i)*time.Second),
+		)
+	}
+
+	body := f.render(t)
+
+	assert.Contains(t, body, ">50 Most Recent Events</h2>")
+	assert.Equal(t, 50, strings.Count(body, `title="Body size"`))
+	assert.NotContains(t, body, "application/x-recent-00")
+	assert.Contains(t, body, "application/x-recent-01")
+	assert.Less(
+		t,
+		strings.Index(body, "application/x-recent-50"),
+		strings.Index(body, "application/x-recent-49"),
+	)
+}
+
+// TestHandleSourceDetail_RecentEventColumns proves a row shows its
+// time relative with the UTC timestamp on hover, its body size,
+// and its processing time once every delivery has finished.
+func TestHandleSourceDetail_RecentEventColumns(t *testing.T) {
+	t.Parallel()
+
+	f := newRecentEventsFixture(t)
+	logTarget := seedTarget(t, f.db, f.webhook.ID, database.TargetTypeLog)
+
+	receivedAt := time.Now().Add(-210 * time.Second).
+		UTC().Truncate(time.Second)
+
+	done := f.event(
+		t, contentTypeJSON, strings.Repeat("x", 2048), receivedAt,
+	)
+	f.attempt(
+		t,
+		f.delivery(t, done, logTarget.ID, database.DeliveryStatusDelivered),
+		0, 1500*time.Millisecond,
+	)
+
+	waiting := f.event(t, "text/plain", "{}", receivedAt)
+	f.delivery(t, waiting, logTarget.ID, database.DeliveryStatusPending)
+
+	body := f.render(t)
+
+	assert.Contains(
+		t, body,
+		`<span title="`+receivedAt.Format(time.DateTime)+
+			` UTC">3 minutes ago</span>`,
+	)
+	assert.Contains(t, body, `<span title="Body size">2.0 kB</span>`)
+	assert.Contains(t, body, ">1.5s</span>")
+	assert.Contains(t, body, ">in progress</span>")
+}
+
+// TestHandleSourceDetail_StatusWithSingleHTTPTarget proves that a
+// webhook with exactly one HTTP target shows, colour-coded, what
+// that target answered for each event. The log target beside it
+// does not count against "exactly one".
+func TestHandleSourceDetail_StatusWithSingleHTTPTarget(t *testing.T) {
+	t.Parallel()
+
+	f := newRecentEventsFixture(t)
+	target := seedTarget(t, f.db, f.webhook.ID, database.TargetTypeHTTP)
+	seedTarget(t, f.db, f.webhook.ID, database.TargetTypeLog)
+
+	now := time.Now()
+
+	for _, code := range []int{204, 302, 404, 503, 0} {
+		dlv := f.delivery(
+			t, f.event(t, contentTypeJSON, "{}", now), target.ID,
+			database.DeliveryStatusDelivered,
+		)
+		f.attempt(t, dlv, code, time.Second)
+	}
+
+	f.delivery(
+		t, f.event(t, contentTypeJSON, "{}", now), target.ID,
+		database.DeliveryStatusPending,
+	)
+	f.event(t, contentTypeJSON, "{}", now)
+
+	// A replay is a newer delivery, and its answer is the one shown.
+	replayed := f.event(t, contentTypeJSON, "{}", now)
+	f.attempt(t, f.delivery(
+		t, replayed, target.ID, database.DeliveryStatusFailed,
+	), 502, time.Second)
+	f.attempt(t, f.deliveryQueuedAt(
+		t, replayed, target.ID, database.DeliveryStatusDelivered,
+		now.Add(time.Minute),
+	), 200, time.Second)
+
+	body := f.render(t)
+
+	assert.Contains(t, body, statusCell("text-green-600", "204"))
+	assert.Contains(t, body, statusCell("text-gray-500", "302"))
+	assert.Contains(t, body, statusCell("text-yellow-600", "404"))
+	assert.Contains(t, body, statusCell("text-red-600", "503"))
+	assert.Contains(t, body, statusCell("text-red-600", "no response"))
+	assert.Contains(t, body, statusCell("text-gray-400", "pending"))
+	assert.Contains(t, body, statusCell("text-gray-400", "not sent"))
+	assert.Contains(t, body, statusCell("text-green-600", "200"))
+	assert.NotContains(t, body, ">502<")
+}
+
+// TestHandleSourceDetail_NoStatusWithoutSingleHTTPTarget proves the
+// status column is absent when the webhook has no HTTP target or
+// more than one.
+func TestHandleSourceDetail_NoStatusWithoutSingleHTTPTarget(
+	t *testing.T,
+) {
+	t.Parallel()
+
+	cases := map[string][]database.TargetType{
+		"none":    {database.TargetTypeLog},
+		"several": {database.TargetTypeHTTP, database.TargetTypeHTTP},
+	}
+
+	for name, types := range cases {
+		t.Run(name, func(t *testing.T) {
+			t.Parallel()
+
+			f := newRecentEventsFixture(t)
+			event := f.event(t, contentTypeJSON, "{}", time.Now())
+
+			for _, tt := range types {
+				target := seedTarget(t, f.db, f.webhook.ID, tt)
+				f.attempt(t, f.delivery(
+					t, event, target.ID,
+					database.DeliveryStatusDelivered,
+				), 200, time.Second)
+			}
+
+			body := f.render(t)
+
+			assert.Contains(t, body, `title="Body size"`)
+			assert.NotContains(t, body, statusTitle)
+		})
+	}
+}
+
+// TestHandleWebhook_RecordsBodySize proves the receiver records the
+// body's size in bytes, not characters, with the event it stores.
+func TestHandleWebhook_RecordsBodySize(t *testing.T) {
+	t.Parallel()
+
+	f := newRecentEventsFixture(t)
+	seedEntrypoint(t, f.db, f.webhook.ID)
+
+	// Two bytes per character.
+	body := strings.Repeat("é", 1024)
+
+	req := httptest.NewRequestWithContext(
+		context.Background(), http.MethodPost, "/webhook/x",
+		strings.NewReader(body),
+	)
+
+	rctx := chi.NewRouteContext()
+	rctx.URLParams.Add("uuid", "ep-"+f.webhook.ID)
+
+	req = req.WithContext(context.WithValue(
+		req.Context(), chi.RouteCtxKey, rctx,
+	))
+
+	w := httptest.NewRecorder()
+	f.h.HandleWebhook().ServeHTTP(w, req)
+
+	require.Equal(t, http.StatusOK, w.Code)
+
+	var stored database.Event
+
+	require.NoError(t, f.webhookDB.First(&stored).Error)
+	assert.Equal(t, int64(2048), stored.BodyBytes)
+}
+
+// TestHandleSourceDetail_FailedLoadIsAnError proves that when the
+// list cannot be loaded the page answers with an error, rather than
+// an empty list claiming the webhook has no events.
+func TestHandleSourceDetail_FailedLoadIsAnError(t *testing.T) {
+	t.Parallel()
+
+	f := newRecentEventsFixture(t)
+	target := seedTarget(t, f.db, f.webhook.ID, database.TargetTypeHTTP)
+
+	f.attempt(t, f.delivery(
+		t, f.event(t, contentTypeJSON, "{}", time.Now()), target.ID,
+		database.DeliveryStatusDelivered,
+	), 200, time.Second)
+
+	// The attempts are the list's last query, so its events and
+	// deliveries have already loaded when it fails.
+	require.NoError(t, f.webhookDB.Exec(
+		"DROP TABLE delivery_results",
+	).Error)
+
+	w := serveSourceDetailPage(t, f.h, f.sess, f.webhook.ID)
+
+	assert.Equal(t, http.StatusInternalServerError, w.Code)
+	assert.NotContains(t, w.Body.String(), "No events received yet.")
+}

internal/handlers/settings.go

@@ -1,130 +0,0 @@
-package handlers
-
-import (
-	"net/http"
-	"net/netip"
-	"strconv"
-	"strings"
-
-	"sneak.berlin/go/webhooker/internal/config"
-)
-
-// notSet is what the Settings page shows for a value that is empty.
-const notSet = "not set"
-
-// settingRow is one line of the Settings page: an environment
-// variable, what it controls, and the value the server loaded for it.
-type settingRow struct {
-	Name        string
-	Description string
-	Value       string
-}
-
-// HandleSettings returns a handler for the read-only Settings page,
-// which lists the configuration the server started with.
-func (h *Handlers) HandleSettings() http.HandlerFunc {
-	return func(w http.ResponseWriter, r *http.Request) {
-		h.renderTemplate(w, r, "settings.html", map[string]any{
-			"Settings": settingRows(h.params.Config),
-		})
-	}
-}
-
-// settingRows lists every field of cfg under the environment variable
-// it is read from, in the order of the README's configuration table.
-// METRICS_PASSWORD and SENTRY_DSN are credentials, so their values
-// never reach the page: only whether they are set.
-func settingRows(cfg *config.Config) []settingRow {
-	metricsUsername := cfg.MetricsUsername
-	if metricsUsername == "" {
-		metricsUsername = notSet
-	}
-
-	return []settingRow{
-		{"WEBHOOKER_ENVIRONMENT", "dev or prod", cfg.Environment},
-		{"PORT", "HTTP listen port", strconv.Itoa(cfg.Port)},
-		{
-			"BIND_ADDRESS",
-			"IP address the HTTP listener binds",
-			cfg.BindAddress,
-		},
-		{
-			"DATA_DIR",
-			"Directory for all SQLite databases",
-			cfg.DataDir,
-		},
-		{
-			"DEBUG",
-			"Enable debug logging",
-			strconv.FormatBool(cfg.Debug),
-		},
-		{
-			"MAINTENANCE_MODE",
-			"Report maintenanceMode: true in the healthcheck JSON. " +
-				"It does not change how any request is served",
-			strconv.FormatBool(cfg.MaintenanceMode),
-		},
-		{
-			"METRICS_USERNAME",
-			"Basic auth username for /metrics",
-			metricsUsername,
-		},
-		{
-			"METRICS_PASSWORD",
-			"Basic auth password for /metrics",
-			setOrNotSet(cfg.MetricsPassword),
-		},
-		{
-			"SENTRY_DSN",
-			"Error reporting DSN. Unset leaves error reporting off",
-			setOrNotSet(cfg.SentryDSN),
-		},
-		{
-			"RETENTION_SWEEP_INTERVAL",
-			"How often the retention reaper and archive sweeper run",
-			cfg.RetentionSweepInterval.String(),
-		},
-		{
-			"SESSION_IDLE_TIMEOUT",
-			"Idle session timeout. Zero or negative disables idle " +
-				"expiry",
-			cfg.SessionIdleTimeout.String(),
-		},
-		{
-			"RECEIVER_RATE_LIMIT",
-			"Receiver requests per minute per IP per entrypoint " +
-				"(10x that per IP across the route)",
-			strconv.Itoa(cfg.ReceiverRateLimit),
-		},
-		{
-			"TRUSTED_PROXIES",
-			"CIDRs whose forwarded headers are trusted",
-			cidrList(cfg.TrustedProxies),
-		},
-		{
-			"ALLOWED_EGRESS_CIDRS",
-			"CIDRs that delivery targets may reach despite the " +
-				"SSRF blocklist",
-			cidrList(cfg.AllowedEgressCIDRs),
-		},
-	}
-}
-
-// setOrNotSet is how the Settings page shows a credential: whether it
-// has a value, never the value itself.
-func setOrNotSet(value string) string {
-	if value == "" {
-		return notSet
-	}
-
-	return "set"
-}
-
-// cidrList renders a CIDR list setting for the Settings page.
-func cidrList(prefixes []netip.Prefix) string {
-	if len(prefixes) == 0 {
-		return "none"
-	}
-
-	return strings.Join(config.PrefixStrings(prefixes), ", ")
-}

internal/handlers/settings_test.go

@@ -1,130 +0,0 @@
-package handlers_test
-
-import (
-	"context"
-	"html"
-	"net/http"
-	"net/http/httptest"
-	"net/netip"
-	"regexp"
-	"testing"
-	"time"
-
-	"github.com/stretchr/testify/assert"
-	"github.com/stretchr/testify/require"
-	"sneak.berlin/go/webhooker/internal/config"
-	"sneak.berlin/go/webhooker/internal/handlers"
-	"sneak.berlin/go/webhooker/internal/session"
-)
-
-// settingsShown renders the Settings page over cfg as a logged-in user
-// and returns the value it shows for each variable name, plus the
-// whole page.
-func settingsShown(
-	t *testing.T, cfg *config.Config,
-) (map[string]string, string) {
-	t.Helper()
-
-	var h *handlers.Handlers
-
-	var sess *session.Session
-
-	app := newTestAppWithConfig(t, cfg, &h, &sess)
-	app.RequireStart()
-
-	t.Cleanup(app.RequireStop)
-
-	req := httptest.NewRequestWithContext(
-		context.Background(), http.MethodGet, "/settings", nil,
-	)
-	for _, c := range authenticatedCookies(t, sess, "id", "admin") {
-		req.AddCookie(c)
-	}
-
-	w := httptest.NewRecorder()
-	h.HandleSettings().ServeHTTP(w, req)
-	require.Equal(t, http.StatusOK, w.Code)
-
-	body := w.Body.String()
-
-	row := regexp.MustCompile(
-		`<code[^>]*>([A-Z_]+)</code>\s*<code[^>]*>([^<]*)</code>`,
-	)
-
-	shown := map[string]string{}
-	for _, match := range row.FindAllStringSubmatch(body, -1) {
-		shown[match[1]] = html.UnescapeString(match[2])
-	}
-
-	return shown, body
-}
-
-func TestSettingsPageShowsLoadedConfiguration(t *testing.T) {
-	t.Parallel()
-
-	const (
-		metricsPassword = "metrics-password-1f9a"
-		sentryKey       = "dsnkey7c2e"
-		sentryDSN       = "https://" + sentryKey + "@errors.example.com/42"
-	)
-
-	cfg := &config.Config{
-		DataDir:                t.TempDir(),
-		Debug:                  true,
-		MaintenanceMode:        true,
-		Environment:            config.EnvironmentDev,
-		MetricsUsername:        "scraper",
-		MetricsPassword:        metricsPassword,
-		Port:                   9123,
-		SentryDSN:              sentryDSN,
-		BindAddress:            "192.0.2.10",
-		RetentionSweepInterval: 17 * time.Minute,
-		SessionIdleTimeout:     3 * time.Hour,
-		ReceiverRateLimit:      77,
-		TrustedProxies: []netip.Prefix{
-			netip.MustParsePrefix("10.1.0.0/16"),
-		},
-		AllowedEgressCIDRs: []netip.Prefix{
-			netip.MustParsePrefix("192.168.5.0/24"),
-			netip.MustParsePrefix("fd00::/8"),
-		},
-	}
-
-	shown, body := settingsShown(t, cfg)
-
-	assert.Equal(t, map[string]string{
-		"WEBHOOKER_ENVIRONMENT":    "dev",
-		"PORT":                     "9123",
-		"BIND_ADDRESS":             "192.0.2.10",
-		"DATA_DIR":                 cfg.DataDir,
-		"DEBUG":                    "true",
-		"MAINTENANCE_MODE":         "true",
-		"METRICS_USERNAME":         "scraper",
-		"METRICS_PASSWORD":         "set",
-		"SENTRY_DSN":               "set",
-		"RETENTION_SWEEP_INTERVAL": "17m0s",
-		"SESSION_IDLE_TIMEOUT":     "3h0m0s",
-		"RECEIVER_RATE_LIMIT":      "77",
-		"TRUSTED_PROXIES":          "10.1.0.0/16",
-		"ALLOWED_EGRESS_CIDRS":     "192.168.5.0/24, fd00::/8",
-	}, shown)
-
-	assert.NotContains(t, body, metricsPassword)
-	assert.NotContains(t, body, sentryKey)
-	assert.Contains(
-		t, body, `href="/settings"`,
-		"the navigation bar links to the page",
-	)
-}
-
-func TestSettingsPageShowsUnsetValues(t *testing.T) {
-	t.Parallel()
-
-	shown, _ := settingsShown(t, &config.Config{DataDir: t.TempDir()})
-
-	assert.Equal(t, "not set", shown["METRICS_USERNAME"])
-	assert.Equal(t, "not set", shown["METRICS_PASSWORD"])
-	assert.Equal(t, "not set", shown["SENTRY_DSN"])
-	assert.Equal(t, "none", shown["TRUSTED_PROXIES"])
-	assert.Equal(t, "none", shown["ALLOWED_EGRESS_CIDRS"])
-}

internal/handlers/source_detail_test.go

@@ -62,6 +62,23 @@ func renderSourceDetailPage(
 ) string {
 	t.Helper()
 
+	w := serveSourceDetailPage(t, h, sess, webhookID)
+
+	require.Equal(t, http.StatusOK, w.Code)
+
+	return w.Body.String()
+}
+
+// serveSourceDetailPage runs the real source detail handler for a
+// webhook and returns its response, whatever its status.
+func serveSourceDetailPage(
+	t *testing.T,
+	h *handlers.Handlers,
+	sess *session.Session,
+	webhookID string,
+) *httptest.ResponseRecorder {
+	t.Helper()
+
 	req := httptest.NewRequestWithContext(
 		context.Background(),
 		http.MethodGet,
@@ -87,9 +104,7 @@ func renderSourceDetailPage(
 	w := httptest.NewRecorder()
 	h.HandleSourceDetail().ServeHTTP(w, req)
 
-	require.Equal(t, http.StatusOK, w.Code)
-
-	return w.Body.String()
+	return w
 }
 
 // TestHandleSourceDetail_MasksSlackWebhookURL is the

internal/handlers/source_management.go

@@ -415,16 +415,23 @@ func (h *Handlers) renderSourceDetail(
 		"webhook_id = ?", webhook.ID,
 	).Find(&targets)
 
-	var events []database.Event
+	var events []RecentEventView
 
 	if h.dbMgr.DBExists(webhook.ID) {
-		webhookDB, dbErr := h.dbMgr.GetDB(webhook.ID)
-		if dbErr == nil {
-			webhookDB.Where(
-				"webhook_id = ?", webhook.ID,
-			).Order("created_at DESC").Limit(
-				recentEventLimit,
-			).Find(&events)
+		webhookDB, err := h.dbMgr.GetDB(webhook.ID)
+		if err != nil {
+			h.serverError(w, "failed to get webhook database", err)
+
+			return
+		}
+
+		events, err = loadRecentEvents(
+			webhookDB, webhook.ID, singleHTTPTargetID(targets),
+		)
+		if err != nil {
+			h.serverError(w, "failed to load recent events", err)
+
+			return
 		}
 	}
 

internal/handlers/webhook.go

@@ -230,6 +230,7 @@ func (s eventSource) event() *database.Event {
 		Method:            s.Method,
 		Headers:           s.HeadersJSON,
 		Body:              string(s.Body),
+		BodyBytes:         int64(len(s.Body)),
 		ContentType:       s.ContentType,
 		ResubmittedFromID: s.ResubmittedFromID,
 	}

internal/server/routes.go

@@ -141,7 +141,6 @@ func (s *Server) setupRoutes() {
 
 	s.setupPageRoutes()
 	s.setupUserRoutes()
-	s.setupSettingsRoutes()
 	s.setupSourceRoutes()
 	s.setupWebhookRoutes()
 }
@@ -182,21 +181,6 @@ func (s *Server) setupUserRoutes() {
 	})
 }
 
-// setupSettingsRoutes serves the Settings page. It is GET only:
-// configuration comes from the environment and nothing here changes
-// it.
-func (s *Server) setupSettingsRoutes() {
-	s.router.Route("/settings", func(r chi.Router) {
-		// MaxBodySize precedes CSRF and RequireAuth deliberately;
-		// see maxFormBodySize for why, and for what it costs.
-		r.Use(s.mw.MaxBodySize(maxFormBodySize))
-		r.Use(s.mw.CSRF())
-		r.Use(s.mw.NoCache())
-		r.Use(s.mw.RequireAuth())
-		r.Get("/", s.h.HandleSettings())
-	})
-}
-
 func (s *Server) setupSourceRoutes() {
 	s.router.Route("/sources", func(r chi.Router) {
 		// MaxBodySize precedes CSRF and RequireAuth deliberately;

internal/server/settings_test.go

@@ -1,22 +0,0 @@
-package server_test
-
-import (
-	"net/http"
-	"testing"
-
-	"github.com/stretchr/testify/assert"
-)
-
-func TestSettingsPageIsBehindLogin(t *testing.T) {
-	t.Parallel()
-
-	env := newTestEnv(t)
-
-	w := env.get("/settings", nil)
-	assert.Equal(t, http.StatusSeeOther, w.Code)
-	assert.Equal(t, "/pages/login", w.Header().Get("Location"))
-
-	w = env.get("/settings", env.authCookies(t, "id", "admin"))
-	assert.Equal(t, http.StatusOK, w.Code)
-	assert.Contains(t, w.Body.String(), "WEBHOOKER_ENVIRONMENT")
-}

templates/navbar.html

@@ -17,7 +17,6 @@
         <div class="hidden md:flex items-center gap-4">
             {{if .User}}
             <a href="/sources" class="btn-text">Webhooks</a>
-            <a href="/settings" class="btn-text">Settings</a>
             <a href="/user/{{.User.Username}}" class="btn-text">
                 <svg class="w-5 h-5 mr-1" fill="currentColor" viewBox="0 0 16 16">
                     <path d="M11 6a3 3 0 1 1-6 0 3 3 0 0 1 6 0z"/>
@@ -40,7 +39,6 @@
         <div class="flex flex-col gap-2">
             {{if .User}}
             <a href="/sources" class="btn-text w-full text-left">Webhooks</a>
-            <a href="/settings" class="btn-text w-full text-left">Settings</a>
             <a href="/user/{{.User.Username}}" class="btn-text w-full text-left">Profile</a>
             <form method="POST" action="/pages/logout">
                 <input type="hidden" name="csrf_token" value="{{.CSRFToken}}">

templates/profile.html

@@ -41,10 +41,6 @@
                     <dt class="w-32 text-sm font-medium text-gray-500">Username</dt>
                     <dd class="text-sm text-gray-900">{{.User.Username}}</dd>
                 </div>
-                <div class="flex">
-                    <dt class="w-32 text-sm font-medium text-gray-500">Account Type</dt>
-                    <dd class="text-sm text-gray-900">Standard User</dd>
-                </div>
             </dl>
         </div>
     </div>

templates/settings.html

@@ -1,24 +0,0 @@
-{{template "base" .}}
-
-{{define "title"}}Settings - Webhooker{{end}}
-
-{{define "content"}}
-<div class="max-w-6xl mx-auto px-6 py-8">
-    <h1 class="text-2xl font-medium text-gray-900">Settings</h1>
-    <p class="text-sm text-gray-500 mt-1 mb-6">The configuration this server started with. It is set in the server's environment and cannot be changed here.</p>
-
-    <div class="card">
-        <div class="divide-y divide-gray-100">
-            {{range .Settings}}
-            <div class="p-4">
-                <div class="flex justify-between items-start gap-4">
-                    <code class="text-sm font-medium text-gray-900">{{.Name}}</code>
-                    <code class="text-sm text-gray-900 break-all">{{.Value}}</code>
-                </div>
-                <p class="text-sm text-gray-500 mt-1">{{.Description}}</p>
-            </div>
-            {{end}}
-        </div>
-    </div>
-</div>
-{{end}}

templates/source_detail.html

@@ -181,18 +181,30 @@
     <!-- Recent Events -->
     <div class="card mt-6">
         <div class="p-4 border-b border-gray-200 flex justify-between items-center">
-            <h2 class="text-lg font-medium text-gray-900">Recent Events</h2>
+            <h2 class="text-lg font-medium text-gray-900">50 Most Recent Events</h2>
             <a href="/source/{{.Webhook.ID}}/logs" class="btn-text text-sm">View All</a>
         </div>
         <div class="divide-y divide-gray-100">
             {{range .Events}}
             <div class="p-4">
-                <div class="flex items-center justify-between">
-                    <div class="flex items-center gap-3">
+                <div class="flex flex-wrap items-center justify-between gap-3">
+                    <div class="flex flex-wrap items-center gap-3">
                         <span class="badge-info">{{.Method}}</span>
-                        <span class="text-sm text-gray-500">{{.ContentType}}</span>
+                        <span class="text-sm text-gray-500 break-all">{{.ContentType}}</span>
+                        {{if .ResubmittedFromID}}
+                        <span class="text-xs text-gray-500" title="This event is a copy of {{.ResubmittedFromID}}">resubmitted copy</span>
+                        {{end}}
+                    </div>
+                    <div class="flex flex-wrap items-center gap-3 text-xs text-gray-400">
+                        <span title="Body size">{{.Size}}</span>
+                        {{if .ProcessingTime}}
+                        <span title="Processing time: how long the slowest delivery took, from being queued to its last attempt">{{.ProcessingTime}}</span>
+                        {{end}}
+                        {{if .Status}}
+                        <span class="font-medium {{.StatusClass}}" title="HTTP status from the HTTP target">{{.Status}}</span>
+                        {{end}}
+                        <span title="{{.ReceivedUTC}}">{{.Received}}</span>
                     </div>
-                    <span class="text-xs text-gray-400">{{.CreatedAt.Format "2006-01-02 15:04:05 UTC"}}</span>
                 </div>
             </div>
             {{else}}