Show each webhook's activity in the webhook list (closes #394)

Each entry in the webhook list now shows when the webhook's last event
arrived, or "No events yet", and how many of its deliveries failed in
the last 24 hours, in red when that is not zero. The entrypoint and
target counts say how many are inactive.

The figures come from the statistics pane's own reads: the event totals
row, which now also gives the event count, shown as events within
retention, instead of counting every stored event, and the pane's query
over the deliveries finished in the last 24 hours. A webhook whose event
database cannot be read says so in its entry rather than showing zeros.

Model: opus-5-5

.dockerignore

@@ -1,14 +1,20 @@
+# .git is sent so the build can derive the version it stamps into the binary
+# (script/version). Its config, which can hold a remote URL carrying a
+# credential and which `git describe` does not need, is left out of a
+# directory context. A context sent as a tar is not filtered by this file, so
+# it carries .git/config unless its sender leaves it out.
+.git/config
+
+# No tracked file may be listed here: git in the build would see it as
+# deleted and mark the version -dirty.
+#
 # .ci-fingerprint is deliberately NOT excluded: it is the CI cache barrier
 # that keeps the check stages from replaying a cached pass. See the lint
 # stage of the Dockerfile.
-.git/
 bin/
 # Extracted from 3p/ by `make assets` inside the build; a host copy is not
 # needed. The tarball in 3p/ must stay in the context.
 static/js/alpine.min.js
-*.md
-LICENSE
-.editorconfig
 .env
 .env.*
 *.db

.gitea/workflows/check.yml

@@ -12,9 +12,8 @@ jobs:
       - name: Checkout
         uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 2024-10-23
         with:
-          # The fingerprint step below needs history to find the last commit
+          # The superseded-status step needs history to walk ancestors (it
-          # that touched the Docker build context, and the superseded-status
+          # aborts on a shallow clone).
-          # step needs it to walk ancestors (it aborts on a shallow clone).
           fetch-depth: 0
 
       - name: Mark superseded run statuses
@@ -28,16 +27,11 @@ jobs:
         run: script/ci-mark-superseded
 
       - name: Fingerprint the build context
-        # `.dockerignore` keeps docs out of the build context, so a docs-only
+        # Writes the hash of the commit being checked into the context, which
-        # commit legitimately replays the whole image from cache and stays
+        # invalidates the `COPY . .` layer of both check stages: a commit
-        # cheap. Every other commit writes a new fingerprint into the context,
+        # that was never linted, format-checked, tested and built cannot
-        # which invalidates the `COPY . .` layer of both check stages: a
+        # report success from cache.
-        # commit that was never linted, formatted-checked, tested and built
+        run: git rev-parse HEAD > .ci-fingerprint
-        # cannot report success from cache.
-        run: |
-          set -eu
-          fp="$(git log -1 --format=%H -- . ':!*.md' ':!LICENSE' ':!.editorconfig')"
-          printf '%s\n' "${fp:-$GITHUB_SHA}" > .ci-fingerprint
 
       - name: Build Docker image (runs make check)
         run: script/cibuild

Dockerfile

@@ -12,8 +12,8 @@ WORKDIR /src
 COPY go.mod go.sum ./
 RUN go mod download
 
-# Copy source code. In CI the context also carries .ci-fingerprint, whose
+# Copy source code. In CI the context also carries .ci-fingerprint, which
-# value changes with every commit that touches the build context (see
+# holds the hash of the commit being checked (see
 # .gitea/workflows/check.yml). That invalidates this layer, so the checks
 # below cannot report success by replaying a cached pass. Do not add it to
 # .dockerignore.
@@ -38,8 +38,13 @@ FROM golang:1.26.1-bookworm@sha256:4465644228bc2857a954b092167e12aa59c006a349228
 COPY --from=lint /src/go.sum /dev/null
 
 # jq is a runtime dependency of script/ci-mark-superseded, which the test
-# suite executes.
+# suite executes. git is what script/version derives the version with.
-RUN apt-get update && apt-get install -y --no-install-recommends make curl ca-certificates jq && rm -rf /var/lib/apt/lists/*
+RUN apt-get update && apt-get install -y --no-install-recommends make curl ca-certificates jq git && rm -rf /var/lib/apt/lists/*
+
+# A build context sent as a tar archive keeps its files' owners, and git
+# refuses to read a checkout owned by another user. Trust this one
+# whoever owns it.
+RUN git config --system --add safe.directory /build
 
 WORKDIR /build
 
@@ -55,14 +60,22 @@ COPY . .
 # from its tarball in 3p/.
 RUN make test
 
-# Version stamped into the binary. .dockerignore excludes .git/, so
+# Version stamped into the binary: the VERSION build arg when one is
-# nothing in this stage can derive it: script/docker resolves it on the
+# given, otherwise what script/version derives from the .git the build
-# host and passes it in. The default is what a bare `docker build .`
+# context carries, so any `docker build .` of a clone stamps its commit.
-# with no --build-arg gets, and it names no tag the tree may not be at.
+# With neither, as from a source tarball, it is "unknown".
 #
 # Declared here, below the test step, so a changed version does not
 # invalidate its cached layer.
-ARG VERSION=unknown
+ARG VERSION
+
+# A context that carries .git must not stamp "unknown": that means git is
+# missing here or could not read the checkout, and the image could not be
+# traced back to its commit.
+RUN if [ -d .git ] && [ "$(make version VERSION="$VERSION")" = unknown ]; then \
+        echo "version is unknown although the build context carries .git" >&2; \
+        exit 1; \
+    fi
 
 RUN make build VERSION="$VERSION"
 

Makefile

@@ -4,12 +4,12 @@
 .DEFAULT_GOAL := check
 
 # Version stamped into the binary. Derived from git by script/version;
-# override it (`make build VERSION=v1.2.3`) where git metadata is
+# override it (`make build VERSION=v1.2.3`) to stamp a given value, which is
-# unavailable, which is how the Dockerfile passes its build arg in.
+# how the Dockerfile passes its build arg in.
 VERSION ?= $(shell script/version)
 
-# An empty override (`make build VERSION=`, or a `--build-arg VERSION=`
+# An empty override (`make build VERSION=`, or the Dockerfile's `make build
-# landing on the Dockerfile's `make build VERSION="$VERSION"`) means unset,
+# VERSION="$VERSION"` when no VERSION build arg was given) means unset,
 # exactly as it does in script/version -- stamping "" would leave the binary
 # reporting no version and the footer back on its "dev" fallback. `override`
 # is required: a plain assignment loses to the command-line definition it

README.md

@@ -1133,13 +1133,29 @@ build itself.
 | Uncommitted changes | the above with a `-dirty` suffix |
 | No git metadata | `unknown` |
 
-`unknown` is what a source tarball or a `docker build .` with no
+The image derives it the same way, from the `.git` that the build
-`--build-arg VERSION=...` reports. `.dockerignore` excludes `.git/`, so
+context carries, so any `docker build .` of a clone, with no build
-the build context carries no git metadata and the image cannot derive
+arguments, stamps the commit it was built from; a shallow clone of one
-the version itself: `script/docker` (and so `make docker`) resolves it
+branch has no tags and stamps the short SHA. `.dockerignore` must
-on the host and passes it in as the `VERSION` build arg. A build that
+therefore leave out neither `.git` nor any tracked file, which git in
-reports `unknown` is a build nobody told what it was; it is not a
+the build would see as deleted, marking the version `-dirty`. It does
-failure, but it cannot be traced back to a commit.
+leave `.git/config`, which can hold a remote URL carrying a credential
+and which `git describe` does not need, out of a directory context. A
+context sent as a tar is not filtered by `.dockerignore`, so it carries
+`.git/config` unless its sender leaves it out; for upaas, that is
+https://git.eeqj.de/sneak/upaas/issues/274. git in the build
+reads the checkout whoever owns its files, since a context sent as a tar
+archive keeps the sender's owners and git otherwise refuses a checkout
+owned by another user. A `VERSION` build arg (`--build-arg VERSION=...`)
+takes precedence; `script/docker` (and so `make docker`) passes the one
+`script/version` resolves on the host. The image build fails if its
+context carries `.git` and the version still comes out `unknown`, which
+means git is missing from the build or could not read the checkout.
+
+`unknown` is what a source tarball, or a `docker build` with no `.git`
+in its context and no `VERSION` build arg, reports. A build that reports
+`unknown` is a build nobody told what it was; it is not a failure, but
+it cannot be traced back to a commit.
 
 `make version` prints what the current checkout would stamp, and
 `make build VERSION=v1.2.3` overrides it. An empty override — from
@@ -1752,7 +1768,7 @@ retries) is individually logged for full observability.
 #### EventTotals and TargetTotals
 
 Running counts in each event database, read by the statistics pane at the
-top of the webhook page. `EventTotals` is one row:
+top of the webhook page and by the webhook list. `EventTotals` is one row:
 
 | Field            | Type      | Description |
 | ---------------- | --------- | ----------- |
@@ -1784,6 +1800,14 @@ target. Its failure percentage for a window is the deliveries that became
 `failed` in it out of all that became `delivered` or `failed` in it, and
 a dash when none did.
 
+The webhook list at `/hooks` shows three of the pane's figures for each
+webhook: its events within retention and its last event, both from
+`EventTotals`, and its deliveries that failed in the last 24 hours,
+counted with the pane's query. It opens each webhook's event database once
+(the handle stays open) and runs those two reads there, so its cost grows
+with the number of webhooks and, for each, with the deliveries that
+finished in the last 24 hours, never with the events stored.
+
 #### Event-tier indexes
 
 These indexes on the per-webhook event databases are declared in the model
@@ -1791,7 +1815,7 @@ tags, so `AutoMigrate` creates them on a fresh database:
 
 | Table              | Columns                     | Serves |
 | ------------------ | --------------------------- | ------ |
-| `deliveries`       | `status`, `deleted_at`, `finished_at`, `target_id` | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status, and the webhook page's statistics, which count each target's deliveries by status and when they finished |
+| `deliveries`       | `status`, `deleted_at`, `finished_at`, `target_id` | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status, and the webhook page's statistics and the webhook list, which count each target's deliveries by status and when they finished |
 | `deliveries`       | `event_id`, `deleted_at`    | The event log, which loads each event's deliveries, and retention, which counts and deletes the deliveries of expired events |
 | `delivery_results` | `delivery_id`, `deleted_at` | The event log, which loads the attempts of a page's deliveries, and retention, which deletes the attempts of expired events |
 | `events`           | `deleted_at`, `created_at`  | The webhook page's statistics, which count recent events |
@@ -2935,13 +2959,15 @@ Components are wired via Uber fx in this order:
 7. `healthcheck.New` — Health check service
 8. `session.New` — Cookie-based session manager (key from database)
 9. `handlers.New` — HTTP handlers
-10. `middleware.New` — HTTP middleware
+10. `metrics.NewRegistry` — The registry `/metrics` serves
-11. `delivery.New` — Event-driven delivery engine
+11. `metrics.New` — The delivery collectors, registered on that registry
-12. `delivery.NewArchiveSweeper` — Periodic pruning of idle archives
+12. `middleware.New` — HTTP middleware
-13. `delivery.Engine` → `delivery.Notifier` — interface bridge
+13. `delivery.New` — Event-driven delivery engine
-14. `delivery.Engine` → `delivery.WebhookEvictor` — interface bridge so
+14. `delivery.NewArchiveSweeper` — Periodic pruning of idle archives
+15. `delivery.Engine` → `delivery.Notifier` — interface bridge
+16. `delivery.Engine` → `delivery.WebhookEvictor` — interface bridge so
     deleting a webhook releases its archive writer
-15. `server.New` — HTTP server and router
+17. `server.New` — HTTP server and router
 
 The server starts via `fx.Invoke(func(*server.Server, *delivery.Engine,
 *database.RetentionReaper, *delivery.ArchiveSweeper) {})`, which
@@ -2984,6 +3010,12 @@ local record instead of nothing. What that placement gives up is
 recovery of a panic in the six entries above it, none of which does
 more than set a header or start a timer.
 
+Each admin page route group (`/pages`, `/user/*`, `/hooks`,
+`/hook/*`) starts with its own **Recoverer** and, if `SENTRY_DSN` is
+set, its own **Sentry** error reporting. That Recoverer answers a panic
+with the `500` error page in the normal layout; the global one keeps
+the plain-text `500` for every other route.
+
 Additionally, form endpoints (`/pages`, `/user/*`, `/hooks`,
 `/hook/*`) apply a **MaxBodySize** middleware that limits
 POST/PUT/PATCH request bodies to 1 MB. It is registered ahead of the
@@ -3227,8 +3259,9 @@ version is fixed independently of the compiler's:
    rebuilds the binary with `CGO_ENABLED=1` and static linking so it
    runs on musl. Both builds go through `make build`, the relink adding
    its `-extldflags` via `GO_LDFLAGS`, so neither can drop the `-X` that
-   stamps the version. The version arrives as the `VERSION` build arg,
+   stamps the version. The version is the `VERSION` build arg if one is
-   since the context has no `.git` (see
+   given, otherwise derived from the `.git` in the context, and the
+   stage fails if a context with `.git` would stamp `unknown` (see
    [Version stamping](#version-stamping)).
 3. **Runtime stage** (`alpine:3.21`) — copies the static binary and
    `deploy/docker-entrypoint.sh`, creates the `/var/lib/webhooker`
@@ -3260,19 +3293,13 @@ A layer cache lets `docker build .` exit 0 in seconds with the lint and
 test stages replayed rather than executed, which would make a green
 check meaningless. The `check` workflow therefore writes
 `.ci-fingerprint` into the build context before building. Its value is
-the hash of the last commit that touched the build context, so:
+the hash of the commit being checked, so every commit, docs-only ones
+and a squash merge whose tree matches an already-built branch included,
+gets a new fingerprint, invalidates the `COPY . .` layer of both check
+stages, and really runs `make fmt-check`, `golangci-lint`, `make test`,
+and `make build`. A run that reports success ran them.
 
-- Any commit that changes code (including a squash merge whose tree
+The module download layer sits above `COPY . .` and stays cached.
-  matches an already-built branch) gets a new fingerprint, invalidates
-  the `COPY . .` layer of both check stages, and really runs
-  `make fmt-check`, `golangci-lint`, `make test`, and `make build`. A
-  run that reports success ran them.
-- A docs-only commit leaves the fingerprint unchanged — `.dockerignore`
-  excludes `*.md`, `LICENSE` and `.editorconfig` from the context
-  anyway — so the image replays from cache and costs seconds.
-
-The module download layer sits above `COPY . .` and stays cached either
-way.
 
 A separate workflow step, run before the fingerprint is written, covers
 a second way the gate lied: Gitea cancels an in-flight run when a newer

cmd/webhooker/main.go

@@ -16,6 +16,7 @@ import (
 	"sneak.berlin/go/webhooker/internal/handlers"
 	"sneak.berlin/go/webhooker/internal/healthcheck"
 	"sneak.berlin/go/webhooker/internal/logger"
+	"sneak.berlin/go/webhooker/internal/metrics"
 	"sneak.berlin/go/webhooker/internal/middleware"
 	"sneak.berlin/go/webhooker/internal/resetpw"
 	"sneak.berlin/go/webhooker/internal/server"
@@ -177,6 +178,10 @@ func newApp() *fx.App {
 			healthcheck.New,
 			session.New,
 			handlers.New,
+			// The registry /metrics serves, and the delivery
+			// collectors registered on it.
+			metrics.NewRegistry,
+			metrics.New,
 			middleware.New,
 			// The one SSRF guard both target-creation validation
 			// and the delivery dialer consult, so they cannot

internal/delivery/engine.go

@@ -148,6 +148,7 @@ type EngineParams struct {
 	DBManager *database.WebhookDBManager
 	Logger    *logger.Logger
 	SSRFGuard *Guard
+	Metrics   *metrics.Set
 }
 
 // Engine processes queued deliveries in the background
@@ -167,10 +168,10 @@ type Engine struct {
 	retryCh    chan Task
 	workers    int
 
-	// mtr is the delivery metric set. Production wires the
+	// mtr is the delivery metric set. Production wires the one
-	// process-wide one; a test can substitute a set registered on
+	// registered on the registry /metrics serves; a test can
-	// a private registry so its assertions are not disturbed by
+	// substitute a set registered on a registry it holds, so it can
-	// deliveries other tests are making at the same time.
+	// gather what its own deliveries recorded.
 	mtr *metrics.Set
 
 	// targets maps each target type to its implementation.
@@ -204,7 +205,7 @@ func New(
 		deliveryCh: make(chan Task, deliveryChannelSize),
 		retryCh:    make(chan Task, retryChannelSize),
 		workers:    defaultWorkers,
-		mtr:        metrics.Default(),
+		mtr:        params.Metrics,
 	}
 
 	e.initTargets(&http.Client{

internal/delivery/export_test.go

@@ -9,6 +9,7 @@ import (
 	"net/url"
 	"time"
 
+	"github.com/prometheus/client_golang/prometheus"
 	"go.uber.org/fx"
 	"gorm.io/gorm"
 	"sneak.berlin/go/webhooker/internal/database"
@@ -40,11 +41,6 @@ const (
 	ExportPendingSweepMinAge = pendingSweepMinAge
 )
 
-// ExportIsBlockedIP exposes isBlockedIP for testing.
-func ExportIsBlockedIP(ip net.IP) bool {
-	return isBlockedIP(ip)
-}
-
 // NewTestGuard builds an SSRF Guard from an explicit egress
 // allowlist, without going through config. Passing no prefixes
 // yields the default guard, which blocks every private/reserved
@@ -70,6 +66,11 @@ func ExportBlockedNetworks() []*net.IPNet {
 	return blockedNetworks
 }
 
+// ExportBlockedPublicNetworks exposes blockedPublicNetworks.
+func ExportBlockedPublicNetworks() []*net.IPNet {
+	return blockedPublicNetworks
+}
+
 // ExportIsForwardableHeader exposes isForwardableHeader.
 func ExportIsForwardableHeader(name string) bool {
 	return isForwardableHeader(name)
@@ -399,7 +400,7 @@ func NewTestEngine(
 		deliveryCh: make(chan Task, deliveryChannelSize),
 		retryCh:    make(chan Task, retryChannelSize),
 		workers:    workers,
-		mtr:        metrics.Default(),
+		mtr:        metrics.New(prometheus.NewRegistry()),
 	}
 	e.initTargets(client)
 
@@ -414,7 +415,7 @@ func NewTestEngineSmallRetry(
 	e := &Engine{
 		log:     log,
 		retryCh: make(chan Task, 1),
-		mtr:     metrics.Default(),
+		mtr:     metrics.New(prometheus.NewRegistry()),
 	}
 	e.initTargets(nil)
 
@@ -437,7 +438,7 @@ func NewTestEngineWithDB(
 		deliveryCh: make(chan Task, deliveryChannelSize),
 		retryCh:    make(chan Task, retryChannelSize),
 		workers:    workers,
-		mtr:        metrics.Default(),
+		mtr:        metrics.New(prometheus.NewRegistry()),
 	}
 	e.initTargets(client)
 
@@ -445,8 +446,7 @@ func NewTestEngineWithDB(
 }
 
 // ExportSetMetrics substitutes the engine's metric set, so a test can
-// assert on collectors registered on a private registry instead of
+// assert on collectors registered on a registry it holds.
-// the process-wide ones every other test is also moving.
 func (e *Engine) ExportSetMetrics(mtr *metrics.Set) {
 	e.mtr = mtr
 }

internal/delivery/metrics_test.go

@@ -35,9 +35,8 @@ const (
 )
 
 // mIsolate gives the setup's engine a metric set registered on a
-// private registry. The process-wide collectors are moved by every
+// registry this test holds, so its exact assertions can gather from
-// other delivery test running in parallel, so exact assertions are
+// it.
-// only possible against a registry this test owns.
 func mIsolate(
 	t *testing.T, s iSetup,
 ) *prometheus.Registry {

internal/delivery/ssrf.go

@@ -25,8 +25,16 @@ var (
 	errNoIPs      = errors.New(
 		"hostname resolved to no IP addresses",
 	)
-	errBlockedIP = errors.New(
+	// ErrBlockedPrivateOrReservedIP reports an address in the
-		"blocked private, reserved or cloud metadata address",
+	// default blocklist's private and reserved ranges,
+	// blockedNetworks.
+	ErrBlockedPrivateOrReservedIP = errors.New(
+		"blocked private or reserved address",
+	)
+	// errBlockedPublicMetadata reports a public address on the
+	// default blocklist, one in blockedPublicNetworks.
+	errBlockedPublicMetadata = errors.New(
+		"blocked cloud metadata address",
 	)
 	errBlockedMetadata = errors.New(
 		"blocked link-local or cloud instance metadata " +
@@ -37,22 +45,32 @@ var (
 	)
 )
 
-// blockedNetworks is the default blocklist: the private and
+// blockedNetworks and blockedPublicNetworks together are the
-// reserved IP ranges, plus the public cloud metadata addresses,
+// default blocklist: the private and reserved IP ranges, plus
-// that are blocked to prevent SSRF attacks. An operator can
+// the public cloud metadata addresses, that are blocked to
-// permit specific blocks out of this set with
+// prevent SSRF attacks. An operator can permit specific blocks
-// ALLOWED_EGRESS_CIDRS; see Guard.
+// out of this set with ALLOWED_EGRESS_CIDRS; see Guard.
 //
-// A public address belongs on the default blocklist only if it
+// blockedNetworks holds the private and reserved IP ranges.
-// hands credentials, user data or bootstrap material to whatever
-// can reach it, without the caller presenting anything. A
-// provider's other public addresses are not refused, since
-// reaching them can be legitimate and no list of them could be
-// complete.
 //
 //nolint:gochecknoglobals // package-level network list is appropriate here
 var blockedNetworks []*net.IPNet
 
+// blockedPublicNetworks holds the default blocklist's public
+// addresses, kept apart from blockedNetworks so that they are
+// refused as cloud metadata addresses, never as private or
+// reserved ones.
+//
+// A public address belongs on the default blocklist only if it
+// hands credentials, user data or bootstrap material to whatever
+// can reach it, without the caller presenting anything; it goes
+// in this list. A provider's other public addresses are not
+// refused, since reaching them can be legitimate and no list of
+// them could be complete.
+//
+//nolint:gochecknoglobals // package-level network list is appropriate here
+var blockedPublicNetworks []*net.IPNet
+
 // alwaysBlockedNetworks are the ranges no configuration can
 // open: the link-local blocks and the cloud instance metadata
 // endpoints that live outside them. Reaching one is credential
@@ -88,8 +106,8 @@ var blockedNetworks []*net.IPNet
 // when it clears both halves. Nothing in this list can be
 // reopened, so putting a public address here leaves the operator
 // no escape hatch at all — the condition ALLOWED_EGRESS_CIDRS
-// exists to remove. Default-block it in blockedNetworks instead,
+// exists to remove. Default-block it in blockedPublicNetworks
-// which an allowlist can override.
+// instead, which an allowlist can override.
 //
 // This is a criterion, not an enumeration of every metadata
 // address in existence.
@@ -130,6 +148,9 @@ func init() {
 		"::1/128",
 		"fc00::/7",
 		"fe80::/10",
+	})
+
+	blockedPublicNetworks = mustParseCIDRs([]string{
 		// Azure WireServer, a public address that serves VM credentials.
 		"168.63.129.16/32",
 	})
@@ -225,13 +246,6 @@ func matchesAny(networks []*net.IPNet, ip net.IP) bool {
 	return false
 }
 
-// isBlockedIP checks whether an IP address falls within
-// the default blocklist, before any operator allowlist is
-// considered.
-func isBlockedIP(ip net.IP) bool {
-	return matchesAny(blockedNetworks, ip)
-}
-
 // Guard makes every SSRF decision in the process.
 //
 // It holds the operator's ALLOWED_EGRESS_CIDRS allowlist and
@@ -332,7 +346,8 @@ func (g *Guard) allows(ip net.IP) bool {
 //     consulted, so no configured CIDR reaches link-local or a
 //     cloud metadata endpoint at a non-public address.
 //  2. The allowlist is consulted next, so a listed private
-//     network becomes reachable.
+//     network, or a listed public address on the default
+//     blocklist, becomes reachable.
 //  3. Everything else keeps the default blocklist's answer.
 func (g *Guard) checkIP(ip net.IP) error {
 	if matchesAny(alwaysBlockedNetworks, ip) {
@@ -345,9 +360,15 @@ func (g *Guard) checkIP(ip net.IP) error {
 		return nil
 	}
 
-	if isBlockedIP(ip) {
+	if matchesAny(blockedNetworks, ip) {
 		return fmt.Errorf(
-			"target IP %s: %w", ip, errBlockedIP,
+			"target IP %s: %w", ip, ErrBlockedPrivateOrReservedIP,
+		)
+	}
+
+	if matchesAny(blockedPublicNetworks, ip) {
+		return fmt.Errorf(
+			"target IP %s: %w", ip, errBlockedPublicMetadata,
 		)
 	}
 

internal/delivery/ssrf_allowlist_test.go

@@ -23,6 +23,10 @@ const (
 	metadataIP  = "169.254.169.254"
 	metadataURL = "http://" + metadataIP + "/latest/meta-data/"
 
+	// linkLocalIPv4 is the IPv4 link-local block, which holds
+	// metadataIP.
+	linkLocalIPv4 = "169.254.0.0/16"
+
 	// loopbackHookURL is a target on this host: blocked by
 	// default, reachable only once an operator allowlists
 	// loopback.
@@ -237,7 +241,7 @@ func linkLocalRefusedCases() []metadataAlwaysRefusedCase {
 		},
 		{
 			name:   "whole link-local block",
-			allow:  "169.254.0.0/16",
+			allow:  linkLocalIPv4,
 			target: metadataURL,
 		},
 		{
@@ -412,6 +416,9 @@ func TestGuardAllowlist_AzureWireServerReopenable(t *testing.T) {
 		"WireServer must be refused by the default blocklist, "+
 			"which an allowlist can override",
 	)
+	require.NotErrorIs(t, err, delivery.ErrBlockedPrivateOrReservedIP,
+		"WireServer is public, not private or reserved",
+	)
 
 	assertDialRefused(t, defaultGuard, target)
 
@@ -496,7 +503,7 @@ func TestAlwaysBlockedNetworks_PinnedSet(t *testing.T) {
 	want := []string{
 		// IPv4 link-local: the 169.254.169.254 metadata
 		// service on AWS, Azure and others.
-		"169.254.0.0/16",
+		linkLocalIPv4,
 		// IPv6 link-local.
 		"fe80::/10",
 		// AWS IPv6 IMDS, inside the ULA space an operator may
@@ -526,6 +533,90 @@ func TestAlwaysBlockedNetworks_PinnedSet(t *testing.T) {
 	assert.Equal(t, want, got)
 }
 
+// TestDefaultBlocklist_PinnedSet pins each list of the default
+// blocklist on its own, the private and reserved ranges in
+// blockedNetworks and the public addresses in
+// blockedPublicNetworks, so moving an entry from one list to the
+// other fails it. For the first address of each entry it then
+// checks that the default guard refuses it, and that listing the
+// entry in ALLOWED_EGRESS_CIDRS opens it unless the unconditional
+// set holds that address.
+func TestDefaultBlocklist_PinnedSet(t *testing.T) {
+	t.Parallel()
+
+	// public marks an entry of blockedPublicNetworks; every other
+	// entry belongs in blockedNetworks.
+	tests := []struct {
+		cidr       string
+		public     bool
+		reopenable bool
+	}{
+		{cidr: "127.0.0.0/8", reopenable: true},
+		{cidr: "10.0.0.0/8", reopenable: true},
+		{cidr: "172.16.0.0/12", reopenable: true},
+		{cidr: "192.168.0.0/16", reopenable: true},
+		{cidr: linkLocalIPv4, reopenable: false},
+		{cidr: "0.0.0.0/8", reopenable: true},
+		{cidr: "100.64.0.0/10", reopenable: true},
+		{cidr: "192.0.0.0/24", reopenable: true},
+		{cidr: "192.0.2.0/24", reopenable: true},
+		{cidr: "198.18.0.0/15", reopenable: true},
+		{cidr: "198.51.100.0/24", reopenable: true},
+		{cidr: "203.0.113.0/24", reopenable: true},
+		{cidr: "224.0.0.0/4", reopenable: true},
+		{cidr: "240.0.0.0/4", reopenable: true},
+		{cidr: "::1/128", reopenable: true},
+		{cidr: "fc00::/7", reopenable: true},
+		{cidr: "fe80::/10", reopenable: false},
+		{cidr: "168.63.129.16/32", public: true, reopenable: true},
+	}
+
+	wantPrivate := make([]string, 0, len(tests))
+	wantPublic := make([]string, 0, len(tests))
+
+	for _, tt := range tests {
+		if tt.public {
+			wantPublic = append(wantPublic, tt.cidr)
+		} else {
+			wantPrivate = append(wantPrivate, tt.cidr)
+		}
+	}
+
+	gotPrivate := make([]string, 0, len(tests))
+	for _, n := range delivery.ExportBlockedNetworks() {
+		gotPrivate = append(gotPrivate, n.String())
+	}
+
+	gotPublic := make([]string, 0, len(tests))
+	for _, n := range delivery.ExportBlockedPublicNetworks() {
+		gotPublic = append(gotPublic, n.String())
+	}
+
+	assert.ElementsMatch(t, wantPrivate, gotPrivate, "blockedNetworks")
+	assert.ElementsMatch(t, wantPublic, gotPublic, "blockedPublicNetworks")
+
+	for _, tt := range tests {
+		t.Run(tt.cidr, func(t *testing.T) {
+			t.Parallel()
+
+			prefix := netip.MustParsePrefix(tt.cidr)
+			ip := net.IP(prefix.Addr().AsSlice())
+
+			require.Error(t,
+				delivery.NewTestGuard().ExportCheckIP(ip),
+				"the default guard must refuse %s", ip,
+			)
+
+			err := delivery.NewTestGuard(prefix).ExportCheckIP(ip)
+			if tt.reopenable {
+				assert.NoError(t, err, "listing %s must open it", tt.cidr)
+			} else {
+				assert.Error(t, err, "listing %s must not open it", tt.cidr)
+			}
+		})
+	}
+}
+
 // requireLoopback fails the test unless rawURL's host is a
 // loopback address, so the allowlist test cannot silently stop
 // exercising a blocked range.

internal/delivery/ssrf_test.go

@@ -10,7 +10,7 @@ import (
 	"sneak.berlin/go/webhooker/internal/delivery"
 )
 
-func TestIsBlockedIP_PrivateRanges(t *testing.T) {
+func TestGuardCheckIP_PrivateRanges(t *testing.T) {
 	t.Parallel()
 
 	tests := []struct {
@@ -56,12 +56,14 @@ func TestIsBlockedIP_PrivateRanges(t *testing.T) {
 				"failed to parse IP %s", tt.ip,
 			)
 
+			refused := delivery.NewTestGuard().ExportCheckIP(ip) != nil
+
 			assert.Equal(t,
 				tt.blocked,
-				delivery.ExportIsBlockedIP(ip),
+				refused,
-				"isBlockedIP(%s) = %v, want %v",
+				"default guard refuses %s = %v, want %v",
 				tt.ip,
-				delivery.ExportIsBlockedIP(ip),
+				refused,
 				tt.blocked,
 			)
 		})

internal/handlers/auth.go

@@ -74,7 +74,7 @@ func (h *Handlers) HandleLoginSubmit() http.HandlerFunc {
 		err := r.ParseForm()
 		if err != nil {
 			h.log.Error("failed to parse form", "error", err)
-			http.Error(w, "Bad request", http.StatusBadRequest)
+			h.renderError(w, r, http.StatusBadRequest)
 
 			return
 		}
@@ -212,11 +212,7 @@ func (h *Handlers) authenticateUser(
 
 	valid, err := database.VerifyPassword(password, user.Password)
 	if err != nil {
-		h.log.Error("failed to verify password", "error", err)
+		h.serverError(w, r, "failed to verify password", err)
-		http.Error(
-			w, "Internal server error",
-			http.StatusInternalServerError,
-		)
 
 		return user, err
 	}
@@ -288,24 +284,14 @@ func (h *Handlers) createAuthenticatedSession(
 ) error {
 	oldSess, err := h.session.Get(r)
 	if err != nil {
-		h.log.Error("failed to get session", "error", err)
+		h.serverError(w, r, "failed to get session", err)
-		http.Error(
-			w, "Internal server error",
-			http.StatusInternalServerError,
-		)
 
 		return err
 	}
 
 	sess, err := h.session.Regenerate(r, w, oldSess)
 	if err != nil {
-		h.log.Error(
+		h.serverError(w, r, "failed to regenerate session", err)
-			"failed to regenerate session", "error", err,
-		)
-		http.Error(
-			w, "Internal server error",
-			http.StatusInternalServerError,
-		)
 
 		return err
 	}
@@ -314,11 +300,7 @@ func (h *Handlers) createAuthenticatedSession(
 
 	err = h.session.Save(r, w, sess)
 	if err != nil {
-		h.log.Error("failed to save session", "error", err)
+		h.serverError(w, r, "failed to save session", err)
-		http.Error(
-			w, "Internal server error",
-			http.StatusInternalServerError,
-		)
 
 		return err
 	}

internal/handlers/delivery_replay.go

@@ -105,9 +105,7 @@ func (h *Handlers) HandleDeliveryReplay() http.HandlerFunc {
 		// middleware, which runs before CSRF parses the form.
 		err := r.ParseForm()
 		if err != nil {
-			http.Error(
+			h.renderError(w, r, http.StatusBadRequest)
-				w, "Bad request", http.StatusBadRequest,
-			)
 
 			return
 		}
@@ -124,14 +122,14 @@ func (h *Handlers) replayDelivery(
 	webhook database.Webhook,
 ) {
 	if !h.dbMgr.DBExists(webhook.ID) {
-		http.NotFound(w, r)
+		h.renderError(w, r, http.StatusNotFound)
 
 		return
 	}
 
 	webhookDB, err := h.dbMgr.GetDB(webhook.ID)
 	if err != nil {
-		h.serverError(w, "failed to get webhook database", err)
+		h.serverError(w, r, "failed to get webhook database", err)
 
 		return
 	}
@@ -173,7 +171,7 @@ func (h *Handlers) loadReplaySource(
 			&original, "id = ?", chi.URLParam(r, "deliveryID"),
 		).Error
 	if err != nil {
-		http.NotFound(w, r)
+		h.renderError(w, r, http.StatusNotFound)
 
 		return nil, false
 	}
@@ -195,7 +193,7 @@ func (h *Handlers) queueReplay(
 	)
 	if err != nil {
 		h.serverError(
-			w, "failed to count in-flight deliveries", err,
+			w, r, "failed to count in-flight deliveries", err,
 		)
 
 		return
@@ -212,7 +210,7 @@ func (h *Handlers) queueReplay(
 	err = webhookDB.
 		First(&event, "id = ?", original.EventID).Error
 	if err != nil {
-		h.serverError(w, "failed to load event for replay", err)
+		h.serverError(w, r, "failed to load event for replay", err)
 
 		return
 	}
@@ -222,7 +220,7 @@ func (h *Handlers) queueReplay(
 	)
 	if err != nil {
 		h.serverError(
-			w, "failed to create replay delivery", err,
+			w, r, "failed to create replay delivery", err,
 		)
 
 		return

internal/handlers/error_page_test.go

@@ -0,0 +1,53 @@
+package handlers_test
+
+import (
+	"context"
+	"html/template"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"sneak.berlin/go/webhooker/internal/handlers"
+)
+
+// TestErrorPage_RenderFailureKeepsStatus proves that an error page
+// which cannot render answers with the status it was reporting, as
+// plain text, and is not attempted again: a page whose own render
+// fails reaches the error page, and the error page failing as well
+// ends there with the 500.
+func TestErrorPage_RenderFailureKeepsStatus(t *testing.T) {
+	t.Parallel()
+
+	var h *handlers.Handlers
+
+	app := newTestApp(t, &h)
+	app.RequireStart()
+
+	t.Cleanup(app.RequireStop)
+
+	// .Status is an int, so asking it for a field fails the render.
+	failing := `{{.Status.Missing}}`
+	h.AddTemplateForTest("error.html", template.Must(
+		template.New("error").Parse(failing),
+	))
+	h.AddTemplateForTest("failing.html", template.Must(
+		template.New("failing").Parse(`{{.Data.Missing}}`),
+	))
+
+	req := httptest.NewRequestWithContext(
+		context.Background(), http.MethodGet, "/", nil,
+	)
+
+	w := httptest.NewRecorder()
+	h.HandleErrorPage(http.StatusNotFound).ServeHTTP(w, req)
+
+	assert.Equal(t, http.StatusNotFound, w.Code)
+	assert.Equal(t, "Not Found\n", w.Body.String())
+
+	w = httptest.NewRecorder()
+	h.RenderTemplateForTest(w, req, "failing.html", 0)
+
+	assert.Equal(t, http.StatusInternalServerError, w.Code)
+	assert.Equal(t, "Internal Server Error\n", w.Body.String())
+}

internal/handlers/event_body.go

@@ -52,7 +52,7 @@ func (h *Handlers) HandleEventBodyDownload() http.HandlerFunc {
 		// steered by a client.
 		eventID, err := uuid.Parse(chi.URLParam(r, "eventID"))
 		if err != nil {
-			http.NotFound(w, r)
+			h.renderError(w, r, http.StatusNotFound)
 
 			return
 		}
@@ -103,21 +103,21 @@ func (h *Handlers) serveEventBody(
 	eventID string,
 ) {
 	if !h.dbMgr.DBExists(webhook.ID) {
-		http.NotFound(w, r)
+		h.renderError(w, r, http.StatusNotFound)
 
 		return
 	}
 
 	webhookDB, err := h.dbMgr.GetDB(webhook.ID)
 	if err != nil {
-		h.serverError(w, "failed to get webhook database", err)
+		h.serverError(w, r, "failed to get webhook database", err)
 
 		return
 	}
 
 	body, found, err := eventBody(webhookDB, webhook.ID, eventID)
 	if err != nil {
-		h.serverError(w, "failed to read event body", err)
+		h.serverError(w, r, "failed to read event body", err)
 
 		return
 	}
@@ -130,7 +130,7 @@ func (h *Handlers) serveEventBody(
 	// row and the whole body is served, or it does not and the
 	// response is a clean 404.
 	if !found {
-		http.NotFound(w, r)
+		h.renderError(w, r, http.StatusNotFound)
 
 		return
 	}

internal/handlers/event_resubmit.go

@@ -99,7 +99,7 @@ func (h *Handlers) HandleEventResubmit() http.HandlerFunc {
 		// middleware, which runs before CSRF parses the form.
 		err := r.ParseForm()
 		if err != nil {
-			http.Error(w, "Bad request", http.StatusBadRequest)
+			h.renderError(w, r, http.StatusBadRequest)
 
 			return
 		}
@@ -120,20 +120,20 @@ func (h *Handlers) resubmitEvent(
 	// alphabet rather than from the request.
 	eventID, err := uuid.Parse(chi.URLParam(r, "eventID"))
 	if err != nil {
-		http.NotFound(w, r)
+		h.renderError(w, r, http.StatusNotFound)
 
 		return
 	}
 
 	if !h.dbMgr.DBExists(webhook.ID) {
-		http.NotFound(w, r)
+		h.renderError(w, r, http.StatusNotFound)
 
 		return
 	}
 
 	webhookDB, err := h.dbMgr.GetDB(webhook.ID)
 	if err != nil {
-		h.serverError(w, "failed to get webhook database", err)
+		h.serverError(w, r, "failed to get webhook database", err)
 
 		return
 	}
@@ -147,7 +147,7 @@ func (h *Handlers) resubmitEvent(
 		webhookDB, webhook.ID, eventID.String(),
 	)
 	if err != nil {
-		h.serverError(w, "failed to load event to resubmit", err)
+		h.serverError(w, r, "failed to load event to resubmit", err)
 
 		return
 	}
@@ -155,7 +155,7 @@ func (h *Handlers) resubmitEvent(
 	// A miss is a 404 whether the event was reaped, belongs to
 	// another webhook, or never existed.
 	if !found {
-		http.NotFound(w, r)
+		h.renderError(w, r, http.StatusNotFound)
 
 		return
 	}
@@ -207,7 +207,7 @@ func (h *Handlers) queueResubmit(
 	// inactive one is skipped rather than refused.
 	targets, err := h.loadActiveTargets(webhook.ID)
 	if err != nil {
-		h.serverError(w, "failed to query targets", err)
+		h.serverError(w, r, "failed to query targets", err)
 
 		return
 	}
@@ -225,7 +225,7 @@ func (h *Handlers) queueResubmit(
 		targets,
 	)
 	if err != nil {
-		h.serverError(w, "failed to store resubmitted event", err)
+		h.serverError(w, r, "failed to store resubmitted event", err)
 
 		return
 	}

internal/handlers/export_test.go

@@ -1,9 +1,11 @@
 package handlers
 
 import (
+	"context"
 	"html/template"
 	"log/slog"
 	"net/http"
+	"net/http/httptest"
 	"time"
 
 	"gorm.io/gorm"
@@ -65,7 +67,7 @@ func (s *Handlers) LoadEventLogViewsForTest(
 	page int,
 ) []EventLogView {
 	views, _, _ := s.loadEventsWithDeliveries(
-		w, webhook, nil, page,
+		w, newRequestForTest(), webhook, nil, page,
 	)
 
 	return views
@@ -94,6 +96,14 @@ func FinishedByTargetForTest(
 	return finishedByTarget(webhookDB, since)
 }
 
+// newRequestForTest is the request the helpers here pass on for
+// callers that have none: it is used only to render the error page.
+func newRequestForTest() *http.Request {
+	return httptest.NewRequestWithContext(
+		context.Background(), http.MethodGet, "/", nil,
+	)
+}
+
 // AddTemplateForTest registers a template under a page name so that
 // the handlers_test package can drive the render path with a
 // template of its own.
@@ -147,5 +157,5 @@ func (s *Handlers) BuildDatabaseTargetConfigForTest(
 	w http.ResponseWriter,
 	expiry string,
 ) (string, error) {
-	return s.buildDatabaseTargetConfig(w, expiry)
+	return s.buildDatabaseTargetConfig(w, newRequestForTest(), expiry)
 }

internal/handlers/handlers.go

@@ -12,6 +12,7 @@ import (
 	"net/http"
 	"sync/atomic"
 
+	"github.com/prometheus/client_golang/prometheus"
 	"go.uber.org/fx"
 	"sneak.berlin/go/webhooker/internal/database"
 	"sneak.berlin/go/webhooker/internal/delivery"
@@ -64,6 +65,8 @@ type HandlersParams struct {
 	Notifier     delivery.Notifier
 	Evictor      delivery.WebhookEvictor
 	SSRFGuard    *delivery.Guard
+	Metrics      *metrics.Set
+	Registry     *prometheus.Registry
 }
 
 // Handlers provides HTTP handler methods for all application
@@ -129,7 +132,7 @@ func New(
 	s.mw = params.Middleware
 	s.notifier = params.Notifier
 	s.evictor = params.Evictor
-	s.mtr = metrics.Default()
+	s.mtr = params.Metrics
 	s.ssrf = params.SSRFGuard
 
 	// Parse all page templates once at startup
@@ -142,6 +145,7 @@ func New(
 		"source_edit.html":   parsePageTemplate("source_edit.html"),
 		"source_logs.html":   parsePageTemplate("source_logs.html"),
 		"target_edit.html":   parsePageTemplate("target_edit.html"),
+		"error.html":         parsePageTemplate("error.html"),
 	}
 
 	lc.Append(fx.Hook{
@@ -153,6 +157,16 @@ func New(
 	return s, nil
 }
 
+// HandleErrorPage returns a handler that answers every request with
+// the error page for status. The router uses it for unknown paths, the
+// CSRF middleware for a refused form, and each admin page route
+// group's recoverer for a panic.
+func (s *Handlers) HandleErrorPage(status int) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		s.renderError(w, r, status)
+	}
+}
+
 func (s *Handlers) respondJSON(
 	w http.ResponseWriter,
 	_ *http.Request,
@@ -170,15 +184,76 @@ func (s *Handlers) respondJSON(
 	}
 }
 
-// serverError logs an error and sends a 500 response.
+// serverError logs an error and answers with the 500 error page.
 func (s *Handlers) serverError(
-	w http.ResponseWriter, msg string, err error,
+	w http.ResponseWriter, r *http.Request, msg string, err error,
 ) {
 	s.log.Error(msg, "error", err)
-	http.Error(
+	s.renderError(w, r, http.StatusInternalServerError)
-		w, "Internal server error",
+}
-		http.StatusInternalServerError,
+
-	)
+// renderError answers with status and the error page: the normal
+// layout, one fixed line explaining the status, and a link back to the
+// webhook list, or to sign-in when nobody is signed in.
+//
+// It renders the page itself rather than through renderTemplate,
+// whose own failure comes here. If the error page cannot render
+// either, the answer is the same status in plain text: never a second
+// attempt, and never a different status.
+func (s *Handlers) renderError(
+	w http.ResponseWriter,
+	r *http.Request,
+	status int,
+) {
+	// The page names the signed-in user, and some error pages are
+	// served outside the routes where NoCache runs.
+	w.Header().Set("Cache-Control", "no-store")
+
+	data := s.pageData(r, map[string]any{
+		"Status":     status,
+		"StatusText": http.StatusText(status),
+		"Message":    errorPageText(status),
+	})
+
+	var buf bytes.Buffer
+
+	err := s.templates["error.html"].Execute(&buf, data)
+	if err != nil {
+		s.log.Error("failed to render error page", "error", err)
+		http.Error(w, http.StatusText(status), status)
+
+		return
+	}
+
+	w.Header().Set("Content-Type", "text/html; charset=utf-8")
+	w.WriteHeader(status)
+
+	_, err = buf.WriteTo(w)
+	if err != nil {
+		s.log.Error("failed to write error page", "error", err)
+	}
+}
+
+// errorPageText is the line the error page shows for status. It is
+// fixed per status, so the page tells the reader no more than the
+// plain-text answers it replaced did.
+func errorPageText(status int) string {
+	switch status {
+	case http.StatusBadRequest:
+		return "The request could not be read."
+	case http.StatusForbidden:
+		return "The request was refused. If it came from a form " +
+			"left open for a long time, reload the page and try " +
+			"again."
+	case http.StatusNotFound:
+		return "There is nothing here. It may have been deleted, " +
+			"or the address may be wrong."
+	case http.StatusServiceUnavailable:
+		return "The server is busy. Please try again in a moment."
+	default: // http.StatusInternalServerError
+		return "Something went wrong on the server. Please try " +
+			"again."
+	}
 }
 
 // UserInfo represents user information for templates
@@ -231,14 +306,17 @@ func (s *Handlers) renderTemplate(
 			"template not found",
 			"template", pageTemplate,
 		)
-		http.Error(
+		s.renderError(w, r, http.StatusInternalServerError)
-			w, "Internal server error",
-			http.StatusInternalServerError,
-		)
 
 		return
 	}
 
+	s.executeTemplate(w, r, tmpl, s.pageData(r, data))
+}
+
+// pageData adds the fields the shared layout renders to a page's own
+// data.
+func (s *Handlers) pageData(r *http.Request, data any) any {
 	userInfo := s.getUserInfo(r)
 	csrfToken := middleware.CSRFToken(r)
 
@@ -252,19 +330,16 @@ func (s *Handlers) renderTemplate(
 		m["User"] = userInfo
 		m["CSRFToken"] = csrfToken
 		m["Version"] = version
-		s.executeTemplate(w, tmpl, m)
 
-		return
+		return m
 	}
 
-	wrapper := templateDataWrapper{
+	return templateDataWrapper{
 		User:      userInfo,
 		CSRFToken: csrfToken,
 		Version:   version,
 		Data:      data,
 	}
-
-	s.executeTemplate(w, tmpl, wrapper)
 }
 
 // executeTemplate renders the template into a buffer and writes to
@@ -277,6 +352,7 @@ func (s *Handlers) renderTemplate(
 // this reason.
 func (s *Handlers) executeTemplate(
 	w http.ResponseWriter,
+	r *http.Request,
 	tmpl *template.Template,
 	data any,
 ) {
@@ -287,10 +363,7 @@ func (s *Handlers) executeTemplate(
 		s.log.Error(
 			"failed to execute template", "error", err,
 		)
-		http.Error(
+		s.renderError(w, r, http.StatusInternalServerError)
-			w, "Internal server error",
-			http.StatusInternalServerError,
-		)
 
 		return
 	}

internal/handlers/handlers_test.go

@@ -20,6 +20,7 @@ import (
 	"sneak.berlin/go/webhooker/internal/handlers"
 	"sneak.berlin/go/webhooker/internal/healthcheck"
 	"sneak.berlin/go/webhooker/internal/logger"
+	"sneak.berlin/go/webhooker/internal/metrics"
 	"sneak.berlin/go/webhooker/internal/middleware"
 	"sneak.berlin/go/webhooker/internal/session"
 )
@@ -109,6 +110,8 @@ func newTestApp(
 			func(r *recordingEvictor) delivery.WebhookEvictor {
 				return r
 			},
+			metrics.NewRegistry,
+			metrics.New,
 			middleware.New,
 			delivery.NewGuard,
 			handlers.New,
@@ -307,10 +310,14 @@ func TestRenderTemplateMidRenderErrorSendsNoPartialBody(t *testing.T) {
 		t, http.StatusInternalServerError, w.Code,
 		"a failed render must report a 500",
 	)
-	assert.Equal(
+	assert.NotContains(
-		t, "Internal server error\n", w.Body.String(),
+		t, w.Body.String(), partialPageMarker,
 		"the response must carry no part of the aborted page",
 	)
+	assert.Contains(
+		t, w.Body.String(), "500 Internal Server Error",
+		"a failed render must answer with the error page",
+	)
 }
 
 func TestBuildDatabaseTargetConfig_Valid(t *testing.T) {

internal/handlers/metrics.go

@@ -0,0 +1,21 @@
+package handlers
+
+import (
+	"net/http"
+
+	"github.com/prometheus/client_golang/prometheus/promhttp"
+)
+
+// HandleMetrics returns the Prometheus scrape handler for the
+// registry built by metrics.NewRegistry, which the HTTP, delivery, Go
+// runtime and process collectors register on. It is what
+// promhttp.Handler builds for the global default registry, including
+// the promhttp_metric_handler_* series that count scrapes, pointed at
+// that registry instead.
+func (s *Handlers) HandleMetrics() http.HandlerFunc {
+	reg := s.params.Registry
+
+	return promhttp.InstrumentMetricHandler(
+		reg, promhttp.HandlerFor(reg, promhttp.HandlerOpts{}),
+	).ServeHTTP
+}

internal/handlers/profile.go

@@ -1,7 +1,6 @@
 package handlers
 
 import (
-	"context"
 	"net/http"
 
 	"github.com/go-chi/chi"
@@ -37,14 +36,14 @@ func (h *Handlers) HandlePasswordChange() http.HandlerFunc {
 		err := r.ParseForm()
 		if err != nil {
 			h.log.Error("failed to parse form", "error", err)
-			http.Error(w, "Bad request", http.StatusBadRequest)
+			h.renderError(w, r, http.StatusBadRequest)
 
 			return
 		}
 
 		successMessage, errorMessage, handled := h.applyPasswordChange(
-			r.Context(),
 			w,
+			r,
 			sessionUsername,
 			// PostFormValue, not FormValue: the credential must
 			// come from the body, never from the query string.
@@ -66,12 +65,12 @@ func (h *Handlers) HandlePasswordChange() http.HandlerFunc {
 // applyPasswordChange verifies the current password and, on success,
 // persists a fresh hash for the user, reusing the same helpers that
 // bootstrap the admin user. It returns the success and error messages
-// to display on the profile page. On an internal failure it writes a
+// to display on the profile page. On an internal failure it writes the
-// 500 response itself and returns handled=false, signalling the caller
+// error page itself and returns handled=false, signalling the caller
 // to stop without re-rendering the page.
 func (h *Handlers) applyPasswordChange(
-	ctx context.Context,
 	w http.ResponseWriter,
+	r *http.Request,
 	username, currentPassword, newPassword, confirmPassword string,
 ) (string, string, bool) {
 	// This endpoint verifies one password and hashes another, at
@@ -79,15 +78,10 @@ func (h *Handlers) applyPasswordChange(
 	// endpoint uses. The bound is per hash, not per endpoint: leaving
 	// this path outside it would leave a hole in it. The slot is held
 	// across both hashes.
-	release, ok := h.mw.BeginPasswordVerification(ctx)
+	release, ok := h.mw.BeginPasswordVerification(r.Context())
 	if !ok {
 		h.log.Warn("password verification capacity exhausted")
-		http.Error(
+		h.renderError(w, r, http.StatusServiceUnavailable)
-			w,
-			"The server is busy verifying credentials. "+
-				"Please try again.",
-			http.StatusServiceUnavailable,
-		)
 
 		return "", "", false
 	}
@@ -103,7 +97,7 @@ func (h *Handlers) applyPasswordChange(
 	).First(&user).Error
 	if err != nil {
 		h.serverError(
-			w, "failed to load user for password change", err,
+			w, r, "failed to load user for password change", err,
 		)
 
 		return "", "", false
@@ -113,7 +107,7 @@ func (h *Handlers) applyPasswordChange(
 		currentPassword, user.Password,
 	)
 	if err != nil {
-		h.serverError(w, "failed to verify password", err)
+		h.serverError(w, r, "failed to verify password", err)
 
 		return "", "", false
 	}
@@ -132,7 +126,7 @@ func (h *Handlers) applyPasswordChange(
 
 	hashedPassword, err := database.HashPassword(newPassword)
 	if err != nil {
-		h.serverError(w, "failed to hash new password", err)
+		h.serverError(w, r, "failed to hash new password", err)
 
 		return "", "", false
 	}
@@ -141,7 +135,7 @@ func (h *Handlers) applyPasswordChange(
 		"password", hashedPassword,
 	).Error
 	if err != nil {
-		h.serverError(w, "failed to update password", err)
+		h.serverError(w, r, "failed to update password", err)
 
 		return "", "", false
 	}
@@ -162,7 +156,7 @@ func (h *Handlers) profileOwnerOrDeny(
 ) (string, string, bool) {
 	requestedUsername := chi.URLParam(r, "username")
 	if requestedUsername == "" {
-		http.NotFound(w, r)
+		h.renderError(w, r, http.StatusNotFound)
 
 		return "", "", false
 	}
@@ -172,7 +166,7 @@ func (h *Handlers) profileOwnerOrDeny(
 	// unexpected retrieval error.
 	sess, err := h.session.Get(r)
 	if err != nil {
-		h.serverError(w, "failed to get session", err)
+		h.serverError(w, r, "failed to get session", err)
 
 		return "", "", false
 	}
@@ -180,10 +174,7 @@ func (h *Handlers) profileOwnerOrDeny(
 	sessionUsername, ok := h.session.GetUsername(sess)
 	if !ok {
 		h.log.Error("authenticated session missing username")
-		http.Error(
+		h.renderError(w, r, http.StatusInternalServerError)
-			w, "Internal server error",
-			http.StatusInternalServerError,
-		)
 
 		return "", "", false
 	}
@@ -191,17 +182,14 @@ func (h *Handlers) profileOwnerOrDeny(
 	sessionUserID, ok := h.session.GetUserID(sess)
 	if !ok {
 		h.log.Error("authenticated session missing user ID")
-		http.Error(
+		h.renderError(w, r, http.StatusInternalServerError)
-			w, "Internal server error",
-			http.StatusInternalServerError,
-		)
 
 		return "", "", false
 	}
 
 	// Only allow users to act on their own profile.
 	if requestedUsername != sessionUsername {
-		http.Error(w, "Forbidden", http.StatusForbidden)
+		h.renderError(w, r, http.StatusForbidden)
 
 		return "", "", false
 	}

internal/handlers/profile_test.go

@@ -128,7 +128,9 @@ func TestUserRoute_Unauthenticated_RedirectedByMiddleware(t *testing.T) {
 
 	var sess *session.Session
 
-	app := newTestApp(t, &log, &cfg, &sess)
+	var h *handlers.Handlers
+
+	app := newTestApp(t, &log, &cfg, &sess, &h)
 	app.RequireStart()
 
 	t.Cleanup(app.RequireStop)
@@ -139,7 +141,7 @@ func TestUserRoute_Unauthenticated_RedirectedByMiddleware(t *testing.T) {
 
 	router := chi.NewRouter()
 	router.Route("/user/{username}", func(r chi.Router) {
-		r.Use(mw.CSRF())
+		r.Use(mw.CSRF(h.HandleErrorPage(http.StatusForbidden)))
 		r.Use(mw.RequireAuth())
 		r.Get("/", func(w http.ResponseWriter, _ *http.Request) {
 			handlerReached = true

internal/handlers/source_list_test.go

@@ -0,0 +1,355 @@
+package handlers_test
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"regexp"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"gorm.io/gorm"
+	"gorm.io/gorm/clause"
+	"sneak.berlin/go/webhooker/internal/database"
+	"sneak.berlin/go/webhooker/internal/handlers"
+	"sneak.berlin/go/webhooker/internal/logger"
+	"sneak.berlin/go/webhooker/internal/session"
+)
+
+// failedHighlight is how the list marks a number of failed deliveries
+// that is not zero.
+const failedHighlight = `class="font-medium text-red-600"`
+
+// listWebhook adds a webhook with the given name, owned by the test
+// user.
+func listWebhook(
+	t *testing.T, db *database.Database, name string,
+) *database.Webhook {
+	t.Helper()
+
+	wh := &database.Webhook{UserID: deleteTestUserID, Name: name}
+	require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
+
+	return wh
+}
+
+// renderWebhookList runs the real webhook list handler as the test user
+// and returns the rendered page.
+func renderWebhookList(
+	t *testing.T, h *handlers.Handlers, sess *session.Session,
+) string {
+	t.Helper()
+
+	cookies := authenticatedCookies(
+		t, sess, deleteTestUserID, deleteTestUsername,
+	)
+
+	w := httptest.NewRecorder()
+	h.HandleSourceList().ServeHTTP(
+		w, getRequest(t, "/hooks", cookies, nil),
+	)
+	require.Equal(t, http.StatusOK, w.Code)
+
+	return w.Body.String()
+}
+
+// listCard returns one webhook's entry in a rendered webhook list, its
+// markup as rendered and its text with the markup taken out and each
+// run of space made one space.
+func listCard(t *testing.T, page, webhookID string) (string, string) {
+	t.Helper()
+
+	_, card, found := strings.Cut(page, `href="/hook/`+webhookID+`"`)
+	require.True(t, found, "the list has no entry for %s", webhookID)
+
+	card, _, _ = strings.Cut(card, "</a>")
+	text := regexp.MustCompile(`<[^>]*>`).ReplaceAllString(card, " ")
+
+	return card, strings.Join(strings.Fields(text), " ")
+}
+
+// receiveEvents posts the given number of events to an entrypoint
+// through the real receiver, and returns the webhook's event database
+// and its events, oldest first.
+func receiveEvents(
+	t *testing.T,
+	h *handlers.Handlers,
+	dbMgr *database.WebhookDBManager,
+	webhookID, path string,
+	count int,
+) (*gorm.DB, []database.Event) {
+	t.Helper()
+
+	router := receiverRouter(h)
+
+	for range count {
+		require.Equal(t, http.StatusOK, postReceiver(t, router, path))
+	}
+
+	webhookDB, err := dbMgr.GetDB(webhookID)
+	require.NoError(t, err)
+
+	events := listEvents(t, webhookDB)
+	require.Len(t, events, count)
+
+	return webhookDB, events
+}
+
+// seedFailingWebhook adds a webhook with two entrypoints, one inactive,
+// and four targets, one inactive. Three events each reach the three
+// active targets. Two deliveries failed in the last 24 hours, one 30
+// hours ago, and one was delivered. It returns the webhook and its
+// newest event.
+func seedFailingWebhook(
+	t *testing.T,
+	h *handlers.Handlers,
+	db *database.Database,
+	dbMgr *database.WebhookDBManager,
+) (*database.Webhook, database.Event) {
+	t.Helper()
+
+	wh := listWebhook(t, db, "failing")
+	path := statsEntrypoint(t, db, wh.ID, true)
+
+	statsEntrypoint(t, db, wh.ID, false)
+
+	first := seedTarget(t, db, wh.ID, database.TargetTypeLog)
+	second := seedTarget(t, db, wh.ID, database.TargetTypeLog)
+
+	seedTarget(t, db, wh.ID, database.TargetTypeLog)
+
+	inactive := seedTarget(t, db, wh.ID, database.TargetTypeLog)
+	require.NoError(t, db.DB().Model(inactive).
+		Update("active", false).Error)
+
+	webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, path, 3)
+	now := time.Now()
+
+	statsFinish(t, webhookDB,
+		statsDelivery(t, webhookDB, events[0].ID, first.ID),
+		database.DeliveryStatusFailed, now.Add(-30*time.Hour))
+	statsFinish(t, webhookDB,
+		statsDelivery(t, webhookDB, events[1].ID, first.ID),
+		database.DeliveryStatusFailed, now.Add(-time.Hour))
+	statsFinish(t, webhookDB,
+		statsDelivery(t, webhookDB, events[2].ID, first.ID),
+		database.DeliveryStatusFailed, now.Add(-time.Minute))
+	statsFinish(t, webhookDB,
+		statsDelivery(t, webhookDB, events[2].ID, second.ID),
+		database.DeliveryStatusDelivered, now.Add(-time.Minute))
+
+	return wh, events[2]
+}
+
+// seedHealthyWebhook adds a webhook with one entrypoint and one target,
+// both active, and two events, both delivered. It returns the webhook
+// and its newest event.
+func seedHealthyWebhook(
+	t *testing.T,
+	h *handlers.Handlers,
+	db *database.Database,
+	dbMgr *database.WebhookDBManager,
+) (*database.Webhook, database.Event) {
+	t.Helper()
+
+	wh := listWebhook(t, db, "healthy")
+	path := statsEntrypoint(t, db, wh.ID, true)
+	target := seedTarget(t, db, wh.ID, database.TargetTypeLog)
+
+	webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, path, 2)
+
+	for _, ev := range events {
+		statsFinish(t, webhookDB,
+			statsDelivery(t, webhookDB, ev.ID, target.ID),
+			database.DeliveryStatusDelivered, time.Now())
+	}
+
+	return wh, events[1]
+}
+
+// lastEventText is how the list shows the arrival of an event.
+func lastEventText(ev database.Event) string {
+	return ev.CreatedAt.UTC().Format("2006-01-02 15:04:05 UTC")
+}
+
+// TestSourceList_ShowsActivityOfEachWebhook checks the figures the list
+// shows for a webhook with recent failures, a healthy one, a new one
+// that has received no event, and one without an event database.
+func TestSourceList_ShowsActivityOfEachWebhook(t *testing.T) {
+	t.Parallel()
+
+	var (
+		h     *handlers.Handlers
+		sess  *session.Session
+		db    *database.Database
+		dbMgr *database.WebhookDBManager
+	)
+
+	app := newTestApp(t, &h, &sess, &db, &dbMgr)
+	app.RequireStart()
+
+	t.Cleanup(app.RequireStop)
+
+	failing, failingNewest := seedFailingWebhook(t, h, db, dbMgr)
+	healthy, healthyNewest := seedHealthyWebhook(t, h, db, dbMgr)
+
+	// Creating a webhook creates its event database.
+	fresh := listWebhook(t, db, "fresh")
+	require.NoError(t, dbMgr.CreateDB(fresh.ID))
+
+	quiet := listWebhook(t, db, "quiet")
+
+	page := renderWebhookList(t, h, sess)
+
+	card, text := listCard(t, page, failing.ID)
+	assert.Contains(t, text, "2 entrypoints, 1 inactive "+
+		"4 targets, 1 inactive "+
+		"3 events within retention "+
+		"Last event "+lastEventText(failingNewest)+" "+
+		"2 failed deliveries in the last 24 hours")
+	assert.Contains(t, card,
+		failedHighlight+">2 failed deliveries in the last 24 hours<")
+
+	card, text = listCard(t, page, healthy.ID)
+	assert.Contains(t, text, "1 entrypoint "+
+		"1 target "+
+		"2 events within retention "+
+		"Last event "+lastEventText(healthyNewest)+" "+
+		"0 failed deliveries in the last 24 hours")
+	assert.NotContains(t, text, "inactive")
+	assert.NotContains(t, card, failedHighlight)
+
+	card, text = listCard(t, page, fresh.ID)
+	assert.Contains(t, text, "0 entrypoints "+
+		"0 targets "+
+		"0 events within retention "+
+		"No events yet "+
+		"0 failed deliveries in the last 24 hours")
+	assert.NotContains(t, card, failedHighlight)
+
+	card, text = listCard(t, page, quiet.ID)
+	assert.Contains(t, text, "0 entrypoints "+
+		"0 targets "+
+		"0 events within retention "+
+		"No events yet "+
+		"0 failed deliveries in the last 24 hours")
+	assert.NotContains(t, card, failedHighlight)
+	assert.False(t, dbMgr.DBExists(quiet.ID),
+		"showing the list must not create an event database")
+}
+
+// TestSourceList_CountsOnlyEventsWithinRetention checks that once
+// retention has removed one of a webhook's three events, the list
+// counts the two still stored.
+func TestSourceList_CountsOnlyEventsWithinRetention(t *testing.T) {
+	t.Parallel()
+
+	var (
+		h     *handlers.Handlers
+		sess  *session.Session
+		db    *database.Database
+		dbMgr *database.WebhookDBManager
+		log   *logger.Logger
+	)
+
+	app := newTestApp(t, &h, &sess, &db, &dbMgr, &log)
+	app.RequireStart()
+
+	t.Cleanup(app.RequireStop)
+
+	wh := &database.Webhook{
+		UserID: deleteTestUserID, Name: "pruned", RetentionDays: 14,
+	}
+	require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
+
+	path := statsEntrypoint(t, db, wh.ID, true)
+	webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, path, 3)
+
+	statsAge(t, webhookDB, events[0].ID, time.Now().Add(-15*24*time.Hour))
+	statsPrune(t, db, dbMgr, log, webhookDB)
+	require.Len(t, listEvents(t, webhookDB), 2)
+
+	_, text := listCard(t, renderWebhookList(t, h, sess), wh.ID)
+	assert.Contains(t, text,
+		"1 entrypoint 0 targets 2 events within retention")
+}
+
+// TestSourceList_LastEventSurvivesPruningEveryEvent checks that once
+// retention has removed every event of a webhook, the list still shows
+// when the last one arrived rather than "No events yet".
+func TestSourceList_LastEventSurvivesPruningEveryEvent(t *testing.T) {
+	t.Parallel()
+
+	var (
+		h     *handlers.Handlers
+		sess  *session.Session
+		db    *database.Database
+		dbMgr *database.WebhookDBManager
+		log   *logger.Logger
+	)
+
+	app := newTestApp(t, &h, &sess, &db, &dbMgr, &log)
+	app.RequireStart()
+
+	t.Cleanup(app.RequireStop)
+
+	wh := &database.Webhook{
+		UserID: deleteTestUserID, Name: "emptied", RetentionDays: 1,
+	}
+	require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
+
+	path := statsEntrypoint(t, db, wh.ID, true)
+	webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, path, 1)
+
+	statsAge(t, webhookDB, events[0].ID, time.Now().Add(-50*time.Hour))
+	statsPrune(t, db, dbMgr, log, webhookDB)
+	require.Empty(t, listEvents(t, webhookDB))
+
+	_, text := listCard(t, renderWebhookList(t, h, sess), wh.ID)
+	assert.Contains(t, text,
+		"0 events within retention "+
+			"Last event "+lastEventText(events[0]))
+	assert.NotContains(t, text, "No events yet")
+}
+
+// TestSourceList_UnreadableEventDatabase checks that a webhook whose
+// event database cannot be read says so in its entry instead of
+// showing zeros, and that the rest of the list is still shown.
+func TestSourceList_UnreadableEventDatabase(t *testing.T) {
+	t.Parallel()
+
+	var (
+		h     *handlers.Handlers
+		sess  *session.Session
+		db    *database.Database
+		dbMgr *database.WebhookDBManager
+	)
+
+	app := newTestApp(t, &h, &sess, &db, &dbMgr)
+	app.RequireStart()
+
+	t.Cleanup(app.RequireStop)
+
+	broken := listWebhook(t, db, "broken")
+	statsEntrypoint(t, db, broken.ID, true)
+
+	brokenDB, err := dbMgr.GetDB(broken.ID)
+	require.NoError(t, err)
+	require.NoError(t,
+		brokenDB.Migrator().DropTable(&database.EventTotals{}))
+
+	quiet := listWebhook(t, db, "quiet")
+
+	page := renderWebhookList(t, h, sess)
+
+	_, text := listCard(t, page, broken.ID)
+	assert.Contains(t, text,
+		"1 entrypoint 0 targets The event figures could not be read.")
+	assert.NotContains(t, text, "events")
+	assert.NotContains(t, text, "failed")
+
+	_, text = listCard(t, page, quiet.ID)
+	assert.Contains(t, text, "No events yet")
+}

internal/handlers/source_management.go

@@ -3,10 +3,12 @@ package handlers
 import (
 	"encoding/json"
 	"errors"
+	"fmt"
 	"net/http"
 	"slices"
 	"strconv"
 	"strings"
+	"time"
 
 	"github.com/go-chi/chi"
 	"github.com/google/uuid"
@@ -20,9 +22,20 @@ import (
 type WebhookListItem struct {
 	database.Webhook
 
-	EntrypointCount int64
+	EntrypointCount         int
-	TargetCount     int64
+	InactiveEntrypointCount int
+	TargetCount             int
+	InactiveTargetCount     int
+
+	// EventCount is how many events the webhook holds, LastEventAt
+	// when the newest arrived (nil before the first), and
+	// FailedLast24Hours how many of its deliveries failed in the last
+	// 24 hours. When the webhook's event database could not be read,
+	// EventsUnreadable is set and these three are not known.
 	EventCount        int64
+	LastEventAt       *time.Time
+	FailedLast24Hours int64
+	EventsUnreadable  bool
 }
 
 // errMissingURL signals that a required URL was not provided.
@@ -149,18 +162,17 @@ func (h *Handlers) HandleSourceList() http.HandlerFunc {
 			"user_id = ?", userID,
 		).Order("created_at DESC").Find(&webhooks).Error
 		if err != nil {
-			h.log.Error(
+			h.serverError(w, r, "failed to list webhooks", err)
-				"failed to list webhooks", "error", err,
-			)
-			http.Error(
-				w, "Internal server error",
-				http.StatusInternalServerError,
-			)
 
 			return
 		}
 
-		items := h.buildWebhookListItems(webhooks)
+		items, err := h.buildWebhookListItems(webhooks)
+		if err != nil {
+			h.serverError(w, r, "failed to list webhooks", err)
+
+			return
+		}
 
 		data := map[string]any{
 			"Webhooks": items,
@@ -170,36 +182,115 @@ func (h *Handlers) HandleSourceList() http.HandlerFunc {
 	}
 }
 
-// buildWebhookListItems builds list items with counts.
+// buildWebhookListItems builds the list's entry for each webhook. It
+// fails when the main database cannot be read. A webhook whose event
+// database cannot be read is marked on its own entry, and the error is
+// logged.
 func (h *Handlers) buildWebhookListItems(
 	webhooks []database.Webhook,
-) []WebhookListItem {
+) ([]WebhookListItem, error) {
 	items := make([]WebhookListItem, len(webhooks))
+	since := time.Now().Add(-longWindow)
 
 	for i := range webhooks {
-		items[i].Webhook = webhooks[i]
+		item := &items[i]
+		item.Webhook = webhooks[i]
 
-		h.db.DB().Model(&database.Entrypoint{}).Where(
+		var err error
-			"webhook_id = ?", webhooks[i].ID,
-		).Count(&items[i].EntrypointCount)
 
-		h.db.DB().Model(&database.Target{}).Where(
+		item.EntrypointCount, item.InactiveEntrypointCount, err =
-			"webhook_id = ?", webhooks[i].ID,
+			h.countWithInactive(&database.Entrypoint{}, item.ID)
-		).Count(&items[i].TargetCount)
+		if err != nil {
+			return nil, err
+		}
 
-		if h.dbMgr.DBExists(webhooks[i].ID) {
+		item.TargetCount, item.InactiveTargetCount, err =
-			webhookDB, err := h.dbMgr.GetDB(
+			h.countWithInactive(&database.Target{}, item.ID)
-				webhooks[i].ID,
+		if err != nil {
+			return nil, err
+		}
+
+		// Opening an event database that does not exist would create
+		// it, and it would hold nothing to count.
+		if !h.dbMgr.DBExists(item.ID) {
+			continue
+		}
+
+		err = h.readListEventFigures(item, since)
+		if err != nil {
+			h.log.Error(
+				"failed to read webhook list figures",
+				"webhook_id", item.ID,
+				"error", err,
 			)
-			if err == nil {
+
-				webhookDB.Model(
+			item.EventsUnreadable = true
-					&database.Event{},
-				).Count(&items[i].EventCount)
-			}
 		}
 	}
 
-	return items
+	return items, nil
+}
+
+// countWithInactive returns how many entrypoints or targets, as model
+// says, a webhook has, and how many of them are inactive.
+func (h *Handlers) countWithInactive(
+	model any, webhookID string,
+) (int, int, error) {
+	var active []bool
+
+	err := h.db.DB().Model(model).
+		Where("webhook_id = ?", webhookID).
+		Pluck("active", &active).Error
+	if err != nil {
+		return 0, 0, fmt.Errorf(
+			"reading active flags of webhook %s: %w", webhookID, err,
+		)
+	}
+
+	inactive := 0
+
+	for _, a := range active {
+		if !a {
+			inactive++
+		}
+	}
+
+	return len(active), inactive, nil
+}
+
+// readListEventFigures fills in the figures the list shows from the
+// webhook's event database, with the statistics pane's own queries:
+// the event count and last arrival from the event totals row, and the
+// deliveries that failed since the given time from the deliveries'
+// status index.
+func (h *Handlers) readListEventFigures(
+	item *WebhookListItem, since time.Time,
+) error {
+	webhookDB, err := h.dbMgr.GetDB(item.ID)
+	if err != nil {
+		return err
+	}
+
+	var totals database.EventTotals
+
+	err = webhookDB.Take(&totals).Error
+	if err != nil {
+		return fmt.Errorf("reading event totals: %w", err)
+	}
+
+	item.EventCount = totals.Events - totals.EventsRemoved
+	item.LastEventAt = totals.LastEventAt
+
+	byTarget, err := finishedByTarget(webhookDB, since)
+	if err != nil {
+		return err
+	}
+
+	for _, f := range byTarget {
+		item.FailedLast24Hours += f.Failed
+	}
+
+	return nil
 }
 
 // HandleSourceCreate shows the form to create a new webhook.
@@ -249,9 +340,7 @@ func (h *Handlers) HandleSourceCreateSubmit() http.HandlerFunc {
 		// middleware, which runs before CSRF parses the form.
 		err := r.ParseForm()
 		if err != nil {
-			http.Error(
+			h.renderError(w, r, http.StatusBadRequest)
-				w, "Bad request", http.StatusBadRequest,
-			)
 
 			return
 		}
@@ -311,7 +400,7 @@ func (h *Handlers) createWebhookWithEntrypoint(
 
 	err := h.commitWebhook(webhook)
 	if err != nil {
-		h.serverError(w, "failed to create webhook", err)
+		h.serverError(w, r, "failed to create webhook", err)
 
 		return
 	}
@@ -388,7 +477,7 @@ func (h *Handlers) HandleSourceDetail() http.HandlerFunc {
 			"id = ? AND user_id = ?", sourceID, userID,
 		).First(&webhook).Error
 		if err != nil {
-			http.NotFound(w, r)
+			h.renderError(w, r, http.StatusNotFound)
 
 			return
 		}
@@ -420,7 +509,7 @@ func (h *Handlers) renderSourceDetail(
 	if h.dbMgr.DBExists(webhook.ID) {
 		webhookDB, err := h.dbMgr.GetDB(webhook.ID)
 		if err != nil {
-			h.serverError(w, "failed to get webhook database", err)
+			h.serverError(w, r, "failed to get webhook database", err)
 
 			return
 		}
@@ -429,7 +518,7 @@ func (h *Handlers) renderSourceDetail(
 			webhookDB, webhook.ID, singleHTTPTargetID(targets),
 		)
 		if err != nil {
-			h.serverError(w, "failed to load recent events", err)
+			h.serverError(w, r, "failed to load recent events", err)
 
 			return
 		}
@@ -483,7 +572,7 @@ func (h *Handlers) HandleSourceEdit() http.HandlerFunc {
 			"id = ? AND user_id = ?", sourceID, userID,
 		).First(&webhook).Error
 		if err != nil {
-			http.NotFound(w, r)
+			h.renderError(w, r, http.StatusNotFound)
 
 			return
 		}
@@ -518,7 +607,7 @@ func (h *Handlers) HandleSourceEditSubmit() http.HandlerFunc {
 			"id = ? AND user_id = ?", sourceID, userID,
 		).First(&webhook).Error
 		if err != nil {
-			http.NotFound(w, r)
+			h.renderError(w, r, http.StatusNotFound)
 
 			return
 		}
@@ -527,9 +616,7 @@ func (h *Handlers) HandleSourceEditSubmit() http.HandlerFunc {
 		// middleware, which runs before CSRF parses the form.
 		err = r.ParseForm()
 		if err != nil {
-			http.Error(
+			h.renderError(w, r, http.StatusBadRequest)
-				w, "Bad request", http.StatusBadRequest,
-			)
 
 			return
 		}
@@ -583,7 +670,7 @@ func (h *Handlers) applyWebhookEdit(
 
 	err := h.db.DB().Save(webhook).Error
 	if err != nil {
-		h.serverError(w, "failed to update webhook", err)
+		h.serverError(w, r, "failed to update webhook", err)
 
 		return
 	}
@@ -613,7 +700,7 @@ func (h *Handlers) HandleSourceDelete() http.HandlerFunc {
 			"id = ? AND user_id = ?", sourceID, userID,
 		).First(&webhook).Error
 		if err != nil {
-			http.NotFound(w, r)
+			h.renderError(w, r, http.StatusNotFound)
 
 			return
 		}
@@ -640,7 +727,7 @@ func (h *Handlers) deleteWebhookResources(
 	// be removed by hand; deleted history cannot be recovered.
 	err := h.commitWebhookDeletion(&webhook)
 	if err != nil {
-		h.serverError(w, "failed to delete webhook", err)
+		h.serverError(w, r, "failed to delete webhook", err)
 
 		return
 	}
@@ -666,7 +753,7 @@ func (h *Handlers) deleteWebhookResources(
 		// redirecting as though everything succeeded: the file
 		// needs removing by hand, and the logged error names it.
 		h.serverError(
-			w, "failed to delete webhook event database", err,
+			w, r, "failed to delete webhook event database", err,
 		)
 
 		return
@@ -810,7 +897,7 @@ func (h *Handlers) ownedWebhook(
 		"id = ? AND user_id = ?", sourceID, userID,
 	).First(&webhook).Error
 	if err != nil {
-		http.NotFound(w, r)
+		h.renderError(w, r, http.StatusNotFound)
 
 		return database.Webhook{}, false
 	}
@@ -832,7 +919,7 @@ func (h *Handlers) HandleSourceLogs() http.HandlerFunc {
 			// Without the map every delivery renders through a
 			// zero redactor, so failing the page is the only
 			// safe answer.
-			h.serverError(w, "failed to load targets", err)
+			h.serverError(w, r, "failed to load targets", err)
 
 			return
 		}
@@ -840,7 +927,7 @@ func (h *Handlers) HandleSourceLogs() http.HandlerFunc {
 		page := h.parsePage(r)
 
 		evts, total, ok := h.loadEventsWithDeliveries(
-			w, webhook, targets, page,
+			w, r, webhook, targets, page,
 		)
 		if !ok {
 			return
@@ -950,6 +1037,7 @@ func (h *Handlers) parsePage(r *http.Request) int {
 // caller must then render nothing further.
 func (h *Handlers) loadEventsWithDeliveries(
 	w http.ResponseWriter,
+	r *http.Request,
 	webhook database.Webhook,
 	targetMap map[string]eventLogTarget,
 	page int,
@@ -963,7 +1051,7 @@ func (h *Handlers) loadEventsWithDeliveries(
 	webhookDB, err := h.dbMgr.GetDB(webhook.ID)
 	if err != nil {
 		h.serverError(
-			w, "failed to get webhook database", err,
+			w, r, "failed to get webhook database", err,
 		)
 
 		return nil, 0, false
@@ -1000,7 +1088,7 @@ func (h *Handlers) loadEventsWithDeliveries(
 	)
 	if err != nil {
 		h.serverError(
-			w, "failed to load delivery attempts", err,
+			w, r, "failed to load delivery attempts", err,
 		)
 
 		return nil, 0, false
@@ -1009,7 +1097,7 @@ func (h *Handlers) loadEventsWithDeliveries(
 	resubmits, err := resubmitCounts(webhookDB, eventIDs)
 	if err != nil {
 		h.serverError(
-			w, "failed to count event resubmissions", err,
+			w, r, "failed to count event resubmissions", err,
 		)
 
 		return nil, 0, false
@@ -1232,7 +1320,7 @@ func (h *Handlers) HandleEntrypointCreate() http.HandlerFunc {
 			"id = ? AND user_id = ?", sourceID, userID,
 		).First(&webhook).Error
 		if err != nil {
-			http.NotFound(w, r)
+			h.renderError(w, r, http.StatusNotFound)
 
 			return
 		}
@@ -1241,9 +1329,7 @@ func (h *Handlers) HandleEntrypointCreate() http.HandlerFunc {
 		// middleware, which runs before CSRF parses the form.
 		err = r.ParseForm()
 		if err != nil {
-			http.Error(
+			h.renderError(w, r, http.StatusBadRequest)
-				w, "Bad request", http.StatusBadRequest,
-			)
 
 			return
 		}
@@ -1259,7 +1345,7 @@ func (h *Handlers) HandleEntrypointCreate() http.HandlerFunc {
 
 		err = h.db.DB().Create(entrypoint).Error
 		if err != nil {
-			h.serverError(w, "failed to create entrypoint", err)
+			h.serverError(w, r, "failed to create entrypoint", err)
 
 			return
 		}
@@ -1290,7 +1376,7 @@ func (h *Handlers) HandleTargetCreate() http.HandlerFunc {
 			"id = ? AND user_id = ?", sourceID, userID,
 		).First(&webhook).Error
 		if err != nil {
-			http.NotFound(w, r)
+			h.renderError(w, r, http.StatusNotFound)
 
 			return
 		}
@@ -1299,9 +1385,7 @@ func (h *Handlers) HandleTargetCreate() http.HandlerFunc {
 		// middleware, which runs before CSRF parses the form.
 		err = r.ParseForm()
 		if err != nil {
-			http.Error(
+			h.renderError(w, r, http.StatusBadRequest)
-				w, "Bad request", http.StatusBadRequest,
-			)
 
 			return
 		}
@@ -1372,7 +1456,7 @@ func (h *Handlers) processTargetCreate(
 
 	err = h.db.DB().Create(target).Error
 	if err != nil {
-		h.serverError(w, "failed to create target", err)
+		h.serverError(w, r, "failed to create target", err)
 
 		return
 	}
@@ -1466,7 +1550,7 @@ func (h *Handlers) buildTargetConfig(
 	case database.TargetTypeSlack:
 		return h.buildSlackTargetConfig(w, r, in.URL)
 	case database.TargetTypeDatabase:
-		return h.buildDatabaseTargetConfig(w, in.Expiry)
+		return h.buildDatabaseTargetConfig(w, r, in.Expiry)
 	case database.TargetTypeLog:
 		return "", nil
 	default:
@@ -1516,7 +1600,7 @@ func (h *Handlers) buildHTTPTargetConfig(
 		return "", err
 	}
 
-	return marshalTargetConfig(w, delivery.HTTPTargetConfig{
+	return h.marshalTargetConfig(w, r, delivery.HTTPTargetConfig{
 		URL:     in.URL,
 		Headers: headers,
 		Timeout: timeout,
@@ -1538,7 +1622,7 @@ func (h *Handlers) buildSlackTargetConfig(
 		return "", err
 	}
 
-	return marshalTargetConfig(w, delivery.SlackTargetConfig{
+	return h.marshalTargetConfig(w, r, delivery.SlackTargetConfig{
 		WebhookURL: targetURL,
 	})
 }
@@ -1578,11 +1662,22 @@ func (h *Handlers) validateTargetURL(
 			"url", delivery.MaskURL(targetURL),
 			"error", err,
 		)
-		http.Error(
+
-			w,
+		msg := "Invalid target URL: " + err.Error()
-			"Invalid target URL: "+err.Error(),
+
-			http.StatusBadRequest,
+		// Only a private or reserved address's refusal says how
-		)
+		// to allow it. Metadata refusals never do: link-local and
+		// the other unconditional metadata addresses cannot be
+		// opened, and the default blocklist's public addresses,
+		// which listing does open, hand out credentials.
+		if errors.Is(err, delivery.ErrBlockedPrivateOrReservedIP) {
+			msg += ". Private and reserved addresses are refused " +
+				"by default; the server's ALLOWED_EGRESS_CIDRS " +
+				"setting allows named networks (see \"Allowing " +
+				"egress to your own network\" in the README)."
+		}
+
+		http.Error(w, msg, http.StatusBadRequest)
 
 		return err
 	}
@@ -1592,16 +1687,14 @@ func (h *Handlers) validateTargetURL(
 
 // marshalTargetConfig serialises a target configuration for storage,
 // writing a 500 itself if it cannot.
-func marshalTargetConfig(
+func (h *Handlers) marshalTargetConfig(
 	w http.ResponseWriter,
+	r *http.Request,
 	cfg any,
 ) (string, error) {
 	configBytes, err := json.Marshal(cfg)
 	if err != nil {
-		http.Error(
+		h.serverError(w, r, "failed to encode target config", err)
-			w, "Internal server error",
-			http.StatusInternalServerError,
-		)
 
 		return "", err
 	}
@@ -1617,6 +1710,7 @@ func marshalTargetConfig(
 // expiry yields an empty config (the keep-forever default).
 func (h *Handlers) buildDatabaseTargetConfig(
 	w http.ResponseWriter,
+	r *http.Request,
 	expiry string,
 ) (string, error) {
 	expiry = strings.TrimSpace(expiry)
@@ -1635,8 +1729,8 @@ func (h *Handlers) buildDatabaseTargetConfig(
 		return "", err
 	}
 
-	return marshalTargetConfig(
+	return h.marshalTargetConfig(
-		w, map[string]any{"expiry": expiry},
+		w, r, map[string]any{"expiry": expiry},
 	)
 }
 
@@ -1690,7 +1784,7 @@ func (h *Handlers) deleteChildResource(
 			"id = ? AND user_id = ?", sourceID, userID,
 		).First(&webhook).Error
 		if err != nil {
-			http.NotFound(w, r)
+			h.renderError(w, r, http.StatusNotFound)
 
 			return
 		}
@@ -1700,11 +1794,7 @@ func (h *Handlers) deleteChildResource(
 			childID, webhook.ID,
 		).Delete(model)
 		if result.Error != nil {
-			h.log.Error(errMsg, "error", result.Error)
+			h.serverError(w, r, errMsg, result.Error)
-			http.Error(
-				w, "Internal server error",
-				http.StatusInternalServerError,
-			)
 
 			return
 		}
@@ -1794,18 +1884,14 @@ func (h *Handlers) toggleChildResource(
 			"id = ? AND user_id = ?", sourceID, userID,
 		).First(&webhook).Error
 		if err != nil {
-			http.NotFound(w, r)
+			h.renderError(w, r, http.StatusNotFound)
 
 			return
 		}
 
 		err = toggleFn(webhook.ID, childID)
 		if err != nil {
-			h.log.Error(errMsg, "error", err)
+			h.serverError(w, r, errMsg, err)
-			http.Error(
-				w, "Internal server error",
-				http.StatusInternalServerError,
-			)
 
 			return
 		}

internal/handlers/target_edit.go

@@ -88,9 +88,7 @@ func (h *Handlers) HandleTargetEditSubmit() http.HandlerFunc {
 		// middleware, which runs before CSRF parses the form.
 		err := r.ParseForm()
 		if err != nil {
-			http.Error(
+			h.renderError(w, r, http.StatusBadRequest)
-				w, "Bad request", http.StatusBadRequest,
-			)
 
 			return
 		}
@@ -157,7 +155,7 @@ func (h *Handlers) applyTargetEdit(
 
 	err = h.db.DB().Save(target).Error
 	if err != nil {
-		h.serverError(w, "failed to update target", err)
+		h.serverError(w, r, "failed to update target", err)
 
 		return
 	}
@@ -220,7 +218,7 @@ func (h *Handlers) ownedTarget(
 		chi.URLParam(r, "targetID"), webhook.ID,
 	).First(&target).Error
 	if err != nil {
-		http.NotFound(w, r)
+		h.renderError(w, r, http.StatusNotFound)
 
 		return database.Webhook{}, nil, false
 	}

internal/handlers/target_private_refusal_test.go

@@ -0,0 +1,116 @@
+package handlers_test
+
+import (
+	"net/http"
+	"net/url"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"sneak.berlin/go/webhooker/internal/database"
+)
+
+// privateRefusalHint is the sentence that tells an operator a private
+// destination is refused on purpose, and how to allow one.
+const privateRefusalHint = "Private and reserved addresses are " +
+	"refused by default; the server's ALLOWED_EGRESS_CIDRS setting " +
+	"allows named networks (see \"Allowing egress to your own " +
+	"network\" in the README)."
+
+// TestTargetRefusal_PrivateDestinationSaysHowToAllowIt covers both
+// target types that take a URL, on add and on edit.
+func TestTargetRefusal_PrivateDestinationSaysHowToAllowIt(
+	t *testing.T,
+) {
+	t.Parallel()
+
+	env := setupSourceTest(t)
+
+	targetTypes := []database.TargetType{
+		database.TargetTypeHTTP,
+		database.TargetTypeSlack,
+	}
+
+	for _, targetType := range targetTypes {
+		t.Run(string(targetType), func(t *testing.T) {
+			t.Parallel()
+
+			webhook := seedWebhookWithRetention(t, env.db, 30)
+			targetsPath := "/hook/" + webhook.ID + "/targets"
+
+			form := url.Values{}
+			form.Set("name", "private")
+			form.Set("type", string(targetType))
+			form.Set("url", editBlockedURL)
+
+			added := serveTarget(
+				env, http.MethodPost, targetsPath, form,
+			)
+			assert.Equal(t, http.StatusBadRequest, added.Code)
+			assert.Contains(
+				t, added.Body.String(), privateRefusalHint,
+			)
+
+			form.Set("url", editOriginalURL)
+
+			created := serveTarget(
+				env, http.MethodPost, targetsPath, form,
+			)
+			require.Equal(
+				t, http.StatusSeeOther, created.Code,
+				created.Body.String(),
+			)
+
+			targets := targetsForWebhook(t, env.db, webhook.ID)
+			require.Len(t, targets, 1)
+
+			form.Set("url", editBlockedURL)
+
+			edited := submitTargetEdit(
+				env, webhook.ID, targets[0].ID, form,
+			)
+			assert.Equal(t, http.StatusBadRequest, edited.Code)
+			assert.Contains(
+				t, edited.Body.String(), privateRefusalHint,
+			)
+		})
+	}
+}
+
+// TestTargetRefusal_MetadataDestinationDoesNotSayHowToAllowIt: no
+// setting opens a link-local address, and Azure's WireServer hands out
+// VM credentials, so neither refusal points at the setting.
+func TestTargetRefusal_MetadataDestinationDoesNotSayHowToAllowIt(
+	t *testing.T,
+) {
+	t.Parallel()
+
+	env := setupSourceTest(t)
+
+	metadataURLs := map[string]string{
+		"link-local": "http://169.254.169.254/latest/meta-data/",
+		"wireserver": "http://168.63.129.16/?comp=versions",
+	}
+
+	for name, metadataURL := range metadataURLs {
+		t.Run(name, func(t *testing.T) {
+			t.Parallel()
+
+			webhook := seedWebhookWithRetention(t, env.db, 30)
+
+			form := url.Values{}
+			form.Set("name", "metadata")
+			form.Set("type", string(database.TargetTypeHTTP))
+			form.Set("url", metadataURL)
+
+			w := serveTarget(
+				env, http.MethodPost,
+				"/hook/"+webhook.ID+"/targets", form,
+			)
+			assert.Equal(t, http.StatusBadRequest, w.Code)
+			assert.NotContains(
+				t, w.Body.String(), privateRefusalHint,
+			)
+		})
+	}
+}

internal/handlers/webhook.go

@@ -88,14 +88,14 @@ func (h *Handlers) processWebhookRequest(
 
 	headersJSON, err := json.Marshal(r.Header)
 	if err != nil {
-		h.serverError(w, "failed to serialize headers", err)
+		h.receiverError(w, "failed to serialize headers", err)
 
 		return
 	}
 
 	targets, err := h.loadActiveTargets(entrypoint.WebhookID)
 	if err != nil {
-		h.serverError(w, "failed to query targets", err)
+		h.receiverError(w, "failed to query targets", err)
 
 		return
 	}
@@ -196,7 +196,7 @@ func (h *Handlers) createAndDeliverEvent(
 		targets,
 	)
 	if err != nil {
-		h.serverError(w, "failed to store webhook event", err)
+		h.receiverError(w, "failed to store webhook event", err)
 
 		return
 	}
@@ -204,6 +204,19 @@ func (h *Handlers) createAndDeliverEvent(
 	h.finishWebhookResponse(w, event, entrypoint, tasks)
 }
 
+// receiverError logs an error and answers the sender with a plain-text
+// 500. The receiver's answers are for programs, so it never sends the
+// error page the web UI uses.
+func (h *Handlers) receiverError(
+	w http.ResponseWriter, msg string, err error,
+) {
+	h.log.Error(msg, "error", err)
+	http.Error(
+		w, "Internal server error",
+		http.StatusInternalServerError,
+	)
+}
+
 // eventSource carries the fields a new event is built from. The
 // receiver fills it from the live request; the resubmit handler fills
 // it from a stored event. Both then go through createAndFanOut, so an

internal/metrics/metrics.go

@@ -3,17 +3,18 @@
 // deliveries are attempted, how they end, how long they take, how
 // deep the queues are, and how many circuit breakers are open.
 //
-// The inbound HTTP metrics come from the go-http-metrics recorder in
+// It also builds the registry the authenticated /metrics route
-// internal/middleware and land on prometheus.DefaultRegisterer. These
+// serves. In production, these collectors, the inbound HTTP metrics
-// collectors register there too, so both surfaces are gathered by the
+// recorded in internal/middleware, and the Go runtime and process
-// one promhttp handler mounted on the authenticated /metrics route.
+// collectors all register on that one registry, never on Prometheus's
+// global default.
 package metrics
 
 import (
-	"sync"
 	"time"
 
 	"github.com/prometheus/client_golang/prometheus"
+	"github.com/prometheus/client_golang/prometheus/collectors"
 	"github.com/prometheus/client_golang/prometheus/promauto"
 	"sneak.berlin/go/webhooker/internal/database"
 )
@@ -57,25 +58,31 @@ var knownTargetTypes = []database.TargetType{
 	database.TargetTypeSlack,
 }
 
-// defaultSet is the process-wide metric set, registered on the same
+// NewRegistry returns the registry /metrics serves, carrying the Go
-// registry the HTTP middleware and the /metrics handler already use.
+// runtime and process collectors that Prometheus's global default
-// It is built on first use rather than in an init so that a test
+// registry carries, so the go_* and process_* series stay in the
-// binary that never touches metrics never registers them.
+// scrape.
 //
-//nolint:gochecknoglobals // one process-wide registration, by design
+// A registry of its own, rather than the global default, is what lets
-var defaultSet = sync.OnceValue(func() *Set {
+// two dependency graphs in one process — two tests, say — each
-	return New(prometheus.DefaultRegisterer)
+// register their collectors without the second registration
-})
+// panicking.
+func NewRegistry() *prometheus.Registry {
+	reg := prometheus.NewRegistry()
+	reg.MustRegister(
+		collectors.NewGoCollector(),
+		collectors.NewProcessCollector(
+			collectors.ProcessCollectorOpts{},
+		),
+	)
 
-// Default returns the process-wide metric set.
+	return reg
-func Default() *Set {
-	return defaultSet()
 }
 
 // Set is one registered group of webhooker's delivery collectors.
-// Production uses the single Default set; tests build their own
+// Production builds one on the registry /metrics serves; tests build
-// against a private registry so assertions are not disturbed by
+// one on a registry of their own so they can gather what their own
-// deliveries other tests are making concurrently.
+// deliveries recorded.
 type Set struct {
 	eventsReceived      prometheus.Counter
 	deliveryAttempts    *prometheus.CounterVec
@@ -93,7 +100,7 @@ type Set struct {
 // New registers a full set of delivery collectors on reg and returns
 // it. It panics if reg already holds them, which is the intended
 // behaviour for a duplicate registration.
-func New(reg prometheus.Registerer) *Set {
+func New(reg *prometheus.Registry) *Set {
 	factory := promauto.With(reg)
 
 	s := &Set{

internal/middleware/csrf.go

@@ -19,7 +19,7 @@ func CSRFToken(r *http.Request) string {
 // key to sign a CSRF cookie and validates a masked token submitted via
 // the "csrf_token" form field (or the "X-CSRF-Token" header) on
 // POST/PUT/PATCH/DELETE requests. Requests with an invalid or missing
-// token receive a 403 Forbidden response.
+// token are logged and answered by forbidden, which must write the 403.
 //
 // The middleware detects the client-facing transport protocol
 // per-request via reqtls.IsTLS, the single TLS predicate the session
@@ -36,7 +36,9 @@ func CSRFToken(r *http.Request) string {
 // Two gorilla/csrf instances are maintained — one with Secure cookies
 // (for TLS) and one without (for plaintext HTTP) — because the
 // csrf.Secure option is set at creation time, not per-request.
-func (m *Middleware) CSRF() func(http.Handler) http.Handler {
+func (m *Middleware) CSRF(
+	forbidden http.Handler,
+) func(http.Handler) http.Handler {
 	csrfErrorHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		// CSRF is registered ahead of RequireAuth on every route
 		// group that uses it, so this WARN is reachable by an
@@ -57,7 +59,7 @@ func (m *Middleware) CSRF() func(http.Handler) http.Handler {
 			"remote_addr", r.RemoteAddr,
 			"reason", csrf.FailureReason(r),
 		)
-		http.Error(w, "Forbidden - invalid CSRF token", http.StatusForbidden)
+		forbidden.ServeHTTP(w, r)
 	})
 
 	key := m.session.GetKey()

internal/middleware/csrf_test.go

@@ -18,6 +18,12 @@ import (
 // csrfCookieName is the gorilla/csrf cookie name.
 const csrfCookieName = "_gorilla_csrf"
 
+// forbidden stands in for the error page the server hands CSRF to
+// answer a refused request with.
+func forbidden(w http.ResponseWriter, _ *http.Request) {
+	w.WriteHeader(http.StatusForbidden)
+}
+
 // csrfGetToken performs a GET request through the CSRF middleware
 // and returns the token and cookies.
 func csrfGetToken(
@@ -98,7 +104,7 @@ func TestCSRF_GETSetsToken(t *testing.T) {
 
 	var gotToken string
 
-	handler := m.CSRF()(http.HandlerFunc(
+	handler := m.CSRF(http.HandlerFunc(forbidden))(http.HandlerFunc(
 		func(_ http.ResponseWriter, r *http.Request) {
 			gotToken = middleware.CSRFToken(r)
 		},
@@ -120,7 +126,7 @@ func TestCSRF_POSTWithValidToken(t *testing.T) {
 	t.Parallel()
 
 	m, _ := testMiddleware(t, config.EnvironmentDev)
-	csrfMW := m.CSRF()
+	csrfMW := m.CSRF(http.HandlerFunc(forbidden))
 
 	getReq := httptest.NewRequestWithContext(
 		context.Background(),
@@ -152,7 +158,7 @@ func csrfPOSTWithoutTokenTest(
 	t.Helper()
 
 	m, _ := testMiddleware(t, env)
-	csrfMW := m.CSRF()
+	csrfMW := m.CSRF(http.HandlerFunc(forbidden))
 
 	// GET to establish the CSRF cookie
 	getHandler := csrfMW(http.HandlerFunc(
@@ -209,7 +215,7 @@ func TestCSRF_POSTWithInvalidToken(t *testing.T) {
 	t.Parallel()
 
 	m, _ := testMiddleware(t, config.EnvironmentDev)
-	csrfMW := m.CSRF()
+	csrfMW := m.CSRF(http.HandlerFunc(forbidden))
 
 	// GET to establish the CSRF cookie
 	getHandler := csrfMW(http.HandlerFunc(
@@ -265,7 +271,7 @@ func TestCSRF_GETDoesNotValidate(t *testing.T) {
 
 	var called bool
 
-	handler := m.CSRF()(http.HandlerFunc(
+	handler := m.CSRF(http.HandlerFunc(forbidden))(http.HandlerFunc(
 		func(_ http.ResponseWriter, _ *http.Request) {
 			called = true
 		},
@@ -328,7 +334,7 @@ func csrfTookStrictPath(
 	t.Helper()
 
 	m, _ := testMiddleware(t, env)
-	csrfMW := m.CSRF()
+	csrfMW := m.CSRF(http.HandlerFunc(forbidden))
 
 	newReq := func(method string) *http.Request {
 		r := httptest.NewRequestWithContext(
@@ -477,7 +483,7 @@ func TestCSRF_ProdMode_PlaintextHTTP_POSTWithValidToken(
 	t.Parallel()
 
 	m, _ := testMiddleware(t, config.EnvironmentProd)
-	csrfMW := m.CSRF()
+	csrfMW := m.CSRF(http.HandlerFunc(forbidden))
 
 	getReq := httptest.NewRequestWithContext(
 		context.Background(),
@@ -517,7 +523,7 @@ func TestCSRF_ProdMode_BehindProxy_POSTWithValidToken(
 	t.Parallel()
 
 	m, _ := testMiddleware(t, config.EnvironmentProd)
-	csrfMW := m.CSRF()
+	csrfMW := m.CSRF(http.HandlerFunc(forbidden))
 
 	getReq := httptest.NewRequestWithContext(
 		context.Background(),
@@ -562,7 +568,7 @@ func TestCSRF_ProdMode_DirectTLS_POSTWithValidToken(
 	t.Parallel()
 
 	m, _ := testMiddleware(t, config.EnvironmentProd)
-	csrfMW := m.CSRF()
+	csrfMW := m.CSRF(http.HandlerFunc(forbidden))
 
 	getReq := httptest.NewRequestWithContext(
 		context.Background(),

internal/middleware/export_test.go

@@ -10,8 +10,7 @@ import (
 
 // MetricsMiddlewareForTest builds the metrics recording middleware
 // against a caller-supplied recorder, so a test can gather from its
-// own Prometheus registry rather than the process-wide default one
+// own Prometheus registry without building a whole Middleware.
-// that Middleware.Metrics uses.
 func MetricsMiddlewareForTest(
 	rec httpmetrics.Recorder,
 ) func(http.Handler) http.Handler {

internal/middleware/logbound_test.go

@@ -260,7 +260,9 @@ func logSites() map[string]logSite {
 			) http.Handler {
 				t.Helper()
 
-				return m.CSRF()(unreachable(t))
+				return m.CSRF(http.HandlerFunc(forbidden))(
+					unreachable(t),
+				)
 			},
 			send:       postNoToken,
 			wantStatus: http.StatusForbidden,

internal/middleware/metrics.go

@@ -7,7 +7,6 @@ import (
 
 	"github.com/go-chi/chi"
 	httpmetrics "github.com/slok/go-http-metrics/metrics"
-	prommetrics "github.com/slok/go-http-metrics/metrics/prometheus"
 	ghmm "github.com/slok/go-http-metrics/middleware"
 	"github.com/slok/go-http-metrics/middleware/std"
 )
@@ -151,17 +150,17 @@ func (r boundedLabelRecorder) AddInflightRequests(
 
 var _ httpmetrics.Recorder = boundedLabelRecorder{}
 
-// Metrics returns middleware that records Prometheus HTTP metrics on
+// Metrics returns middleware that records Prometheus HTTP metrics
-// the default registry, which is the one the /metrics route gathers.
+// with the Middleware's one recorder, which New builds on the registry
+// the /metrics route serves and NewForTest on a registry of its own.
+// Every call reuses that recorder, so any number of routers can
+// install it.
 func (s *Middleware) Metrics() func(http.Handler) http.Handler {
-	return metricsMiddleware(
+	return metricsMiddleware(s.metricsRecorder)
-		prommetrics.NewRecorder(prommetrics.Config{}),
-	)
 }
 
 // metricsMiddleware builds the recording middleware against a given
-// recorder, so tests can gather from a registry of their own instead
+// recorder, so tests can gather from a registry of their own.
-// of the process-wide default.
 func metricsMiddleware(
 	rec httpmetrics.Recorder,
 ) func(http.Handler) http.Handler {

internal/middleware/metrics_test.go

@@ -57,9 +57,8 @@ const (
 // Server.setupWebhookRoutes inside it. That ordering is the whole
 // defect, so a test that flattens it would prove nothing.
 //
-// The recorder writes to a registry of the test's own rather than the
+// The recorder writes to a registry of the test's own, so each test
-// process-wide default one, so each test observes only its own
+// observes only its own traffic.
-// traffic.
 func metricsTestRouter(
 	t *testing.T,
 	receiverLimit int,
@@ -455,3 +454,29 @@ func TestMetrics_StatusAndSizeStillRecorded(t *testing.T) {
 		"the interceptor must still count written bytes",
 	)
 }
+
+// TestMetrics_WorksOnNewForTestMiddleware pins that a Middleware built
+// by NewForTest has a recorder of its own: its Metrics() serves a
+// request instead of panicking, and a second one does not collide
+// with the first.
+func TestMetrics_WorksOnNewForTestMiddleware(t *testing.T) {
+	t.Parallel()
+
+	log := slog.New(slog.DiscardHandler)
+	cfg := &config.Config{Environment: "prod"}
+	ok := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = w.Write([]byte(okBody))
+	})
+
+	for range 2 {
+		h := middleware.NewForTest(log, cfg, nil).Metrics()(ok)
+
+		req := httptest.NewRequestWithContext(
+			t.Context(), http.MethodGet, okRoute, nil,
+		)
+		w := httptest.NewRecorder()
+		h.ServeHTTP(w, req)
+
+		assert.Equal(t, http.StatusOK, w.Code)
+	}
+}

internal/middleware/middleware.go

@@ -14,6 +14,9 @@ import (
 	"github.com/go-chi/chi"
 	"github.com/go-chi/chi/middleware"
 	"github.com/go-chi/cors"
+	"github.com/prometheus/client_golang/prometheus"
+	httpmetrics "github.com/slok/go-http-metrics/metrics"
+	prommetrics "github.com/slok/go-http-metrics/metrics/prometheus"
 	"go.uber.org/fx"
 	"sneak.berlin/go/webhooker/internal/config"
 	"sneak.berlin/go/webhooker/internal/globals"
@@ -153,6 +156,7 @@ type MiddlewareParams struct {
 	Globals  *globals.Globals
 	Config   *config.Config
 	Session  *session.Session
+	Registry *prometheus.Registry
 }
 
 // Middleware provides HTTP middleware for logging, CORS, auth, and
@@ -162,6 +166,14 @@ type Middleware struct {
 	params  *MiddlewareParams
 	session *session.Session
 
+	// metricsRecorder records the inbound HTTP metrics. New builds
+	// it on the registry /metrics serves, NewForTest on a registry
+	// of its own. Either way it is built once per Middleware and
+	// Metrics reuses it, because building it registers its
+	// collectors, and a second registration on the same registry
+	// panics.
+	metricsRecorder httpmetrics.Recorder
+
 	// loginGuard counts failed credential verifications and bounds
 	// concurrent password hashing. It is built on first use so that
 	// every construction path gets one; see guard().
@@ -180,6 +192,9 @@ func New(
 	s.params = &params
 	s.log = params.Logger.Get()
 	s.session = params.Session
+	s.metricsRecorder = prommetrics.NewRecorder(
+		prommetrics.Config{Registry: params.Registry},
+	)
 
 	return s, nil
 }

internal/middleware/recoverer.go

@@ -109,7 +109,8 @@ func (w *recoverResponseWriter) Unwrap() http.ResponseWriter {
 
 // Recoverer returns middleware that turns a handler panic into one
 // structured ERROR record and a 500, rather than a dropped
-// connection.
+// connection. The 500 is page when page is not nil, and plain text
+// when it is nil or when page panics before writing anything.
 //
 // It replaces chi's middleware.Recoverer, which does neither on a
 // current Go release. chi v1.5.5's pretty-printer scans the stack for
@@ -136,9 +137,13 @@ func (w *recoverResponseWriter) Unwrap() http.ResponseWriter {
 //
 // Unlike http.Error on its own, it deletes any Set-Cookie the handler
 // set before panicking, because a request that failed must not hand
-// the client a credential; every other header is left to http.Error.
+// the client a credential. It touches no other header: when page
+// answers, every other header the handler set goes out with it, apart
+// from any page sets itself; otherwise they are left to http.Error.
 // See https://git.eeqj.de/sneak/webhooker/issues/193.
-func (s *Middleware) Recoverer() func(http.Handler) http.Handler {
+func (s *Middleware) Recoverer(
+	page http.Handler,
+) func(http.Handler) http.Handler {
 	return func(next http.Handler) http.Handler {
 		return http.HandlerFunc(func(
 			w http.ResponseWriter,
@@ -171,6 +176,14 @@ func (s *Middleware) Recoverer() func(http.Handler) http.Handler {
 
 				rw.Header().Del("Set-Cookie")
 
+				if page != nil {
+					s.servePage(rw, r, page)
+				}
+
+				if rw.committed {
+					return
+				}
+
 				http.Error(
 					rw,
 					http.StatusText(
@@ -185,6 +198,27 @@ func (s *Middleware) Recoverer() func(http.Handler) http.Handler {
 	}
 }
 
+// servePage answers with page. A panic in page itself is logged and
+// recovered here, so the Recoverer can still send its plain 500.
+func (s *Middleware) servePage(
+	w http.ResponseWriter,
+	r *http.Request,
+	page http.Handler,
+) {
+	defer func() {
+		rvr := recover()
+		if rvr != nil {
+			s.log.Error("error page panic",
+				"panic", logfield.Truncate(
+					fmt.Sprint(rvr), maxPanicValueBytes,
+				),
+			)
+		}
+	}()
+
+	page.ServeHTTP(w, r)
+}
+
 // logPanic writes the record. Every field it can grow is truncated to
 // a fixed budget, so MaxPanicLogLineBytes holds.
 //

internal/middleware/recoverer_test.go

@@ -76,7 +76,7 @@ func newRecovererProbe(
 	// Logging outside so the recovered 500 is the status it records.
 	router.Use(chimw.RequestID)
 	router.Use(m.Logging())
-	router.Use(m.Recoverer())
+	router.Use(m.Recoverer(nil))
 	router.Get("/probe", handler)
 
 	serverErrors := new(bytes.Buffer)
@@ -637,7 +637,7 @@ func TestRecovererKeepsResponseControllerWorking(t *testing.T) {
 
 	m, _ := capturingMiddleware(t)
 
-	handler := m.Recoverer()(http.HandlerFunc(
+	handler := m.Recoverer(nil)(http.HandlerFunc(
 		func(w http.ResponseWriter, _ *http.Request) {
 			_, _ = w.Write([]byte("chunk"))
 
@@ -672,3 +672,59 @@ func TestRecovererKeepsResponseControllerWorking(t *testing.T) {
 	assert.Equal(t, http.StatusOK, resp.StatusCode)
 	assert.Equal(t, "chunk", string(body))
 }
+
+// TestRecovererAnswersWithThePage covers a recoverer given a page:
+// the panic is logged as before, and the 500 is that page.
+func TestRecovererAnswersWithThePage(t *testing.T) {
+	t.Parallel()
+
+	m, logs := capturingMiddleware(t)
+
+	page := http.HandlerFunc(
+		func(w http.ResponseWriter, _ *http.Request) {
+			w.WriteHeader(http.StatusInternalServerError)
+			_, _ = w.Write([]byte("the error page"))
+		},
+	)
+
+	w := httptest.NewRecorder()
+	m.Recoverer(page)(http.HandlerFunc(panicProbe)).ServeHTTP(
+		w, httptest.NewRequestWithContext(
+			t.Context(), http.MethodGet, "/", nil,
+		),
+	)
+
+	assert.Equal(t, http.StatusInternalServerError, w.Code)
+	assert.Equal(t, "the error page", w.Body.String())
+	assert.Contains(t, logs.String(), `"msg":"handler panic"`)
+	assert.Contains(t, logs.String(), panicMarker)
+}
+
+// TestRecovererFallsBackWhenThePagePanics covers a page that panics
+// before writing anything: both panics are logged, and the client
+// still gets the plain 500.
+func TestRecovererFallsBackWhenThePagePanics(t *testing.T) {
+	t.Parallel()
+
+	m, logs := capturingMiddleware(t)
+
+	const pagePanic = "QQERRORPAGEPANICQQ"
+
+	page := http.HandlerFunc(
+		func(http.ResponseWriter, *http.Request) {
+			panic(pagePanic)
+		},
+	)
+
+	w := httptest.NewRecorder()
+	m.Recoverer(page)(http.HandlerFunc(panicProbe)).ServeHTTP(
+		w, httptest.NewRequestWithContext(
+			t.Context(), http.MethodGet, "/", nil,
+		),
+	)
+
+	assert.Equal(t, http.StatusInternalServerError, w.Code)
+	assert.Equal(t, "Internal Server Error\n", w.Body.String())
+	assert.Contains(t, logs.String(), panicMarker)
+	assert.Contains(t, logs.String(), pagePanic)
+}

internal/middleware/testing.go

@@ -3,12 +3,17 @@ package middleware
 import (
 	"log/slog"
 
+	"github.com/prometheus/client_golang/prometheus"
+	prommetrics "github.com/slok/go-http-metrics/metrics/prometheus"
 	"sneak.berlin/go/webhooker/internal/config"
 	"sneak.berlin/go/webhooker/internal/session"
 )
 
 // NewForTest creates a Middleware with the minimum dependencies
 // needed for testing. This bypasses the fx lifecycle.
+//
+// Its metrics recorder writes to a fresh registry of its own, so
+// Metrics() works on it and two of them never collide.
 func NewForTest(
 	log *slog.Logger,
 	cfg *config.Config,
@@ -20,5 +25,8 @@ func NewForTest(
 			Config: cfg,
 		},
 		session: sess,
+		metricsRecorder: prommetrics.NewRecorder(
+			prommetrics.Config{Registry: prometheus.NewRegistry()},
+		),
 	}
 }

internal/resetpw/resetpw_test.go

@@ -24,6 +24,7 @@ import (
 	"sneak.berlin/go/webhooker/internal/handlers"
 	"sneak.berlin/go/webhooker/internal/healthcheck"
 	"sneak.berlin/go/webhooker/internal/logger"
+	"sneak.berlin/go/webhooker/internal/metrics"
 	"sneak.berlin/go/webhooker/internal/middleware"
 	"sneak.berlin/go/webhooker/internal/resetpw"
 	"sneak.berlin/go/webhooker/internal/session"
@@ -163,6 +164,8 @@ func newServerApp(
 			session.New,
 			func() delivery.Notifier { return &noopNotifier{} },
 			func() delivery.WebhookEvictor { return &noopEvictor{} },
+			metrics.NewRegistry,
+			metrics.New,
 			middleware.New,
 			delivery.NewGuard,
 			handlers.New,

internal/server/error_page_test.go

@@ -0,0 +1,222 @@
+package server_test
+
+import (
+	"context"
+	"net/http"
+	"net/http/httptest"
+	"net/url"
+	"strconv"
+	"testing"
+
+	"github.com/getsentry/sentry-go"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"sneak.berlin/go/webhooker/internal/config"
+	"sneak.berlin/go/webhooker/internal/server"
+)
+
+// The link back the error page offers: to the webhook list for a
+// signed-in user, to sign-in for anyone else.
+const (
+	backToWebhooks = `<a href="/hooks" class="btn-secondary">` +
+		`Back to webhooks</a>`
+	backToSignIn = `<a href="/pages/login" class="btn-primary">` +
+		`Sign in</a>`
+)
+
+// assertErrorPage checks that w is the error page for status, in the
+// normal layout, offering link.
+func assertErrorPage(
+	t *testing.T,
+	w *httptest.ResponseRecorder,
+	status int,
+	link string,
+) {
+	t.Helper()
+
+	body := w.Body.String()
+
+	assert.Equal(t, status, w.Code)
+	assert.Equal(
+		t, "text/html; charset=utf-8", w.Header().Get("Content-Type"),
+	)
+	assert.Equal(t, "no-store", w.Header().Get("Cache-Control"))
+	assert.Contains(t, body, `<nav class="app-bar"`)
+	assert.Contains(
+		t, body, strconv.Itoa(status)+" "+http.StatusText(status),
+	)
+	assert.Contains(t, body, link)
+}
+
+func TestErrorPage_DeletedWebhook(t *testing.T) {
+	t.Parallel()
+
+	env := newTestEnv(t)
+
+	userID, _ := env.seedUser(t, "owner", "somepassword")
+	cookies := env.authCookies(t, userID, "owner")
+
+	wh := env.seedWebhook(t, userID)
+	require.NoError(t, env.db.DB().Delete(wh).Error)
+
+	w := env.get("/hook/"+wh.ID, cookies)
+
+	assertErrorPage(t, w, http.StatusNotFound, backToWebhooks)
+}
+
+func TestErrorPage_DeletedTarget(t *testing.T) {
+	t.Parallel()
+
+	env := newTestEnv(t)
+
+	userID, _ := env.seedUser(t, "owner", "somepassword")
+	cookies := env.authCookies(t, userID, "owner")
+
+	wh := env.seedWebhook(t, userID)
+	tgt := env.seedTarget(t, wh.ID)
+	require.NoError(t, env.db.DB().Delete(tgt).Error)
+
+	w := env.get(
+		"/hook/"+wh.ID+"/targets/"+tgt.ID+"/edit", cookies,
+	)
+
+	assertErrorPage(t, w, http.StatusNotFound, backToWebhooks)
+}
+
+func TestErrorPage_UnknownPath(t *testing.T) {
+	t.Parallel()
+
+	env := newTestEnv(t)
+
+	userID, _ := env.seedUser(t, "owner", "somepassword")
+	cookies := env.authCookies(t, userID, "owner")
+
+	assertErrorPage(
+		t, env.get("/no-such-page", nil),
+		http.StatusNotFound, backToSignIn,
+	)
+
+	// Outside every route group there is no form token, so the
+	// page leaves out the logout form rather than offer one that
+	// would be refused.
+	w := env.get("/no-such-page", cookies)
+	assertErrorPage(t, w, http.StatusNotFound, backToWebhooks)
+	assert.NotContains(t, w.Body.String(), `action="/pages/logout"`)
+
+	// Inside a route group the page has a token, and logout works.
+	wh := env.seedWebhook(t, userID)
+	w = env.get("/hook/"+wh.ID+"/no-such-page", cookies)
+	assertErrorPage(t, w, http.StatusNotFound, backToWebhooks)
+	assert.Contains(t, w.Body.String(), `action="/pages/logout"`)
+}
+
+func TestErrorPage_BadCSRFToken(t *testing.T) {
+	t.Parallel()
+
+	env := newTestEnv(t)
+
+	form := url.Values{}
+	form.Set("username", "someone")
+	form.Set("password", "irrelevant")
+	form.Set("csrf_token", "not-a-token")
+
+	assertErrorPage(
+		t, env.post("/pages/login", form, nil),
+		http.StatusForbidden, backToSignIn,
+	)
+
+	userID, _ := env.seedUser(t, "owner", "somepassword")
+	cookies := env.authCookies(t, userID, "owner")
+	wh := env.seedWebhook(t, userID)
+
+	edit := url.Values{}
+	edit.Set("name", "renamed")
+
+	assertErrorPage(
+		t, env.post("/hook/"+wh.ID+"/edit", edit, cookies),
+		http.StatusForbidden, backToWebhooks,
+	)
+}
+
+// TestErrorPage_PanicOnAdminPage sends a panicking handler in an
+// admin page route group through the real router, with error
+// tracking on: the client gets the 500 error page, and the tracker
+// still gets the panic, once. The same panic outside the admin page
+// route groups keeps the plain 500.
+func TestErrorPage_PanicOnAdminPage(t *testing.T) {
+	t.Parallel()
+
+	env := newTestEnv(t)
+
+	transport := &captureTransport{}
+
+	opts := server.SentryClientOptionsForTest(
+		"https://public@sentry.invalid/1", "webhooker-test",
+	)
+	opts.Transport = transport
+
+	client, err := sentry.NewClient(opts)
+	require.NoError(t, err)
+
+	serve := func(router http.Handler, path string) *httptest.ResponseRecorder {
+		req := httptest.NewRequestWithContext(
+			sentry.SetHubOnContext(
+				context.Background(),
+				sentry.NewHub(client, sentry.NewScope()),
+			),
+			http.MethodGet, path, nil,
+		)
+
+		w := httptest.NewRecorder()
+		router.ServeHTTP(w, req)
+
+		return w
+	}
+
+	w := serve(
+		server.NewRouterWithPageProbeForTest(
+			env.log.Get(), env.cfg, env.mw, env.hnd,
+			true, panicProbeHandler,
+		),
+		server.PageProbePattern,
+	)
+	assertErrorPage(t, w, http.StatusInternalServerError, backToSignIn)
+
+	w = serve(
+		server.NewRouterWithProbeForTest(
+			env.log.Get(), env.cfg, env.mw, env.hnd,
+			true, panicProbeHandler,
+		),
+		server.ProbePattern,
+	)
+	assert.Equal(t, http.StatusInternalServerError, w.Code)
+	assert.Equal(t, "Internal Server Error\n", w.Body.String())
+
+	require.Len(t, transport.events, 2)
+
+	for _, event := range transport.events {
+		assert.Contains(t, marshalEvent(t, event), panicProbeMarker)
+	}
+}
+
+// TestErrorPage_ReceiverStaysPlain pins that the error page is for
+// the web UI only: a sender posting to an entrypoint that does not
+// exist still gets the plain-text answer.
+func TestErrorPage_ReceiverStaysPlain(t *testing.T) {
+	t.Parallel()
+
+	// newTestEnv leaves the receiver rate limit at zero, which
+	// refuses every request before it reaches the receiver.
+	env := newTestEnvWithConfig(t, &config.Config{
+		DataDir:           t.TempDir(),
+		Environment:       config.EnvironmentDev,
+		ReceiverRateLimit: 10,
+	})
+
+	w := env.post(
+		"/h/0b8f3c1e-7d2a-4e6b-9f15-3a9c2d4e6f70", url.Values{}, nil,
+	)
+
+	assert.Equal(t, http.StatusNotFound, w.Code)
+	assert.Equal(t, "404 page not found\n", w.Body.String())
+}

internal/server/export_test.go

@@ -5,6 +5,7 @@ import (
 	"net/http"
 
 	"github.com/getsentry/sentry-go"
+	"github.com/go-chi/chi"
 	"sneak.berlin/go/webhooker/internal/config"
 	"sneak.berlin/go/webhooker/internal/handlers"
 	"sneak.berlin/go/webhooker/internal/middleware"
@@ -101,3 +102,39 @@ func NewRouterWithProbeForTest(
 
 	return s.router
 }
+
+// PageProbePattern is where NewRouterWithPageProbeForTest serves its
+// probe: inside the /pages route group, the admin page group a
+// request reaches without signing in.
+const PageProbePattern = "/pages/probe"
+
+// NewRouterWithPageProbeForTest is NewRouterWithProbeForTest with the
+// probe added to the /pages route group once SetupRoutes has built
+// it, so the probe runs behind that group's own middleware exactly as
+// the group's real routes do.
+func NewRouterWithPageProbeForTest(
+	log *slog.Logger,
+	cfg *config.Config,
+	mw *middleware.Middleware,
+	h *handlers.Handlers,
+	sentryEnabled bool,
+	probe http.HandlerFunc,
+) http.Handler {
+	s := &Server{
+		log:    log,
+		mw:     mw,
+		h:      h,
+		params: ServerParams{Config: cfg},
+	}
+	s.sentryEnabled.Store(sentryEnabled)
+	s.SetupRoutes()
+
+	for _, route := range s.router.Routes() {
+		pages, ok := route.SubRoutes.(chi.Router)
+		if ok && route.Pattern == "/pages/*" {
+			pages.Get("/probe", probe)
+		}
+	}
+
+	return s.router
+}

internal/server/routes.go

@@ -7,7 +7,6 @@ import (
 	sentryhttp "github.com/getsentry/sentry-go/http"
 	"github.com/go-chi/chi"
 	"github.com/go-chi/chi/middleware"
-	"github.com/prometheus/client_golang/prometheus/promhttp"
 	"sneak.berlin/go/webhooker/static"
 )
 
@@ -15,9 +14,10 @@ import (
 // bytes) for form POST endpoints. 1 MB is generous for any form
 // submission while preventing abuse from oversized payloads.
 //
-// Every route group below installs MaxBodySize(maxFormBodySize) as
+// The four admin page route groups below (/pages, /user/{username},
-// its FIRST middleware, ahead of both CSRF and RequireAuth. Both
+// /hooks and /hook/{sourceID}) install MaxBodySize(maxFormBodySize)
-// orderings are deliberate.
+// right after their recoverer and error reporting, ahead of both CSRF
+// and RequireAuth. Both orderings are deliberate.
 //
 // Ahead of CSRF because gorilla/csrf parses the form. The cap has to
 // be installed before anything reads the body, or the parse runs
@@ -46,6 +46,14 @@ const requestTimeout = 60 * time.Second
 // server's router.
 func (s *Server) SetupRoutes() {
 	s.router = chi.NewRouter()
+
+	// An unknown path gets the error page. Registered before the
+	// global middleware, because chi wraps a not-found handler in the
+	// middleware already on its router, which would then run twice.
+	// The route groups below wrap it in their own middleware the same
+	// way; running theirs twice is harmless.
+	s.router.NotFound(s.h.HandleErrorPage(http.StatusNotFound))
+
 	s.setupGlobalMiddleware()
 	s.setupRoutes()
 }
@@ -69,23 +77,33 @@ func (s *Server) setupGlobalMiddleware() {
 	// Panic recovery, deliberately here rather than first. It has to
 	// run inside every middleware that observes the response, so the
 	// 500 it writes is the status the access log records and the
-	// metrics count, and outside the sentryhttp handler below, whose
+	// metrics count, and outside the sentryhttp handler, whose
 	// Repanic option needs something further out to catch what it
 	// re-raises. chi's own middleware.Recoverer held the first slot
 	// until it was measured: on a current Go release it crashes
 	// inside its stack pretty-printer instead of recovering, so the
 	// connection dropped and the original panic was never reported.
 	// See https://git.eeqj.de/sneak/webhooker/issues/187.
-	s.router.Use(s.mw.Recoverer())
+	s.recoverPanics(s.router, nil)
+}
+
+// recoverPanics installs on r the recoverer, answering a panic with
+// page (a plain 500 when page is nil), and inside it the Sentry error
+// reporting (if SENTRY_DSN is set). Repanic is true so panics still
+// bubble up to the recoverer.
+//
+// Each admin page route group installs its own, with the error page,
+// as its first middleware. A panic there is logged, reported and
+// answered inside the group and never reaches the global recoverer,
+// which keeps the plain 500 for every other route.
+func (s *Server) recoverPanics(r chi.Router, page http.Handler) {
+	r.Use(s.mw.Recoverer(page))
 
-	// Sentry error reporting (if SENTRY_DSN is set). Repanic is
-	// true so panics still bubble up to the Recoverer middleware
-	// registered immediately above.
 	if s.sentryEnabled.Load() {
 		sentryHandler := sentryhttp.New(sentryhttp.Options{
 			Repanic: true,
 		})
-		s.router.Use(sentryHandler.Handle)
+		r.Use(sentryHandler.Handle)
 	}
 }
 
@@ -130,12 +148,7 @@ func (s *Server) setupRoutes() {
 	if s.params.Config.MetricsAuthEnabled() {
 		s.router.Group(func(r chi.Router) {
 			r.Use(s.mw.MetricsAuth())
-			r.Get(
+			r.Get("/metrics", s.h.HandleMetrics())
-				"/metrics",
-				http.HandlerFunc(
-					promhttp.Handler().ServeHTTP,
-				),
-			)
 		})
 	}
 
@@ -147,10 +160,13 @@ func (s *Server) setupRoutes() {
 
 func (s *Server) setupPageRoutes() {
 	s.router.Route("/pages", func(r chi.Router) {
+		s.recoverPanics(
+			r, s.h.HandleErrorPage(http.StatusInternalServerError),
+		)
 		// MaxBodySize precedes CSRF and RequireAuth deliberately;
 		// see maxFormBodySize for why, and for what it costs.
 		r.Use(s.mw.MaxBodySize(maxFormBodySize))
-		r.Use(s.mw.CSRF())
+		r.Use(s.mw.CSRF(s.h.HandleErrorPage(http.StatusForbidden)))
 		r.Use(s.mw.NoCache())
 
 		// The login POST carries no pre-emptive rate limiter. Behind
@@ -169,10 +185,13 @@ func (s *Server) setupPageRoutes() {
 
 func (s *Server) setupUserRoutes() {
 	s.router.Route("/user/{username}", func(r chi.Router) {
+		s.recoverPanics(
+			r, s.h.HandleErrorPage(http.StatusInternalServerError),
+		)
 		// MaxBodySize precedes CSRF and RequireAuth deliberately;
 		// see maxFormBodySize for why, and for what it costs.
 		r.Use(s.mw.MaxBodySize(maxFormBodySize))
-		r.Use(s.mw.CSRF())
+		r.Use(s.mw.CSRF(s.h.HandleErrorPage(http.StatusForbidden)))
 		r.Use(s.mw.NoCache())
 		r.Use(s.mw.RequireAuth())
 		r.Get("/", s.h.HandleProfile())
@@ -184,10 +203,13 @@ func (s *Server) setupUserRoutes() {
 
 func (s *Server) setupSourceRoutes() {
 	s.router.Route("/hooks", func(r chi.Router) {
+		s.recoverPanics(
+			r, s.h.HandleErrorPage(http.StatusInternalServerError),
+		)
 		// MaxBodySize precedes CSRF and RequireAuth deliberately;
 		// see maxFormBodySize for why, and for what it costs.
 		r.Use(s.mw.MaxBodySize(maxFormBodySize))
-		r.Use(s.mw.CSRF())
+		r.Use(s.mw.CSRF(s.h.HandleErrorPage(http.StatusForbidden)))
 		r.Use(s.mw.NoCache())
 		r.Use(s.mw.RequireAuth())
 		r.Get("/", s.h.HandleSourceList())
@@ -196,10 +218,13 @@ func (s *Server) setupSourceRoutes() {
 	})
 
 	s.router.Route("/hook/{sourceID}", func(r chi.Router) {
+		s.recoverPanics(
+			r, s.h.HandleErrorPage(http.StatusInternalServerError),
+		)
 		// MaxBodySize precedes CSRF and RequireAuth deliberately;
 		// see maxFormBodySize for why, and for what it costs.
 		r.Use(s.mw.MaxBodySize(maxFormBodySize))
-		r.Use(s.mw.CSRF())
+		r.Use(s.mw.CSRF(s.h.HandleErrorPage(http.StatusForbidden)))
 		r.Use(s.mw.NoCache())
 		r.Use(s.mw.RequireAuth())
 		r.Get("/", s.h.HandleSourceDetail())

internal/server/routes_test.go

@@ -24,6 +24,7 @@ import (
 	"sneak.berlin/go/webhooker/internal/handlers"
 	"sneak.berlin/go/webhooker/internal/healthcheck"
 	"sneak.berlin/go/webhooker/internal/logger"
+	"sneak.berlin/go/webhooker/internal/metrics"
 	"sneak.berlin/go/webhooker/internal/middleware"
 	"sneak.berlin/go/webhooker/internal/server"
 	"sneak.berlin/go/webhooker/internal/session"
@@ -113,6 +114,8 @@ func newTestEnvWithConfig(
 			session.New,
 			func() delivery.Notifier { return &noopNotifier{} },
 			func() delivery.WebhookEvictor { return &noopEvictor{} },
+			metrics.NewRegistry,
+			metrics.New,
 			middleware.New,
 			delivery.NewGuard,
 			handlers.New,
@@ -1478,3 +1481,46 @@ func TestMetricsRouteUnmountedOnHalfSetConfig(t *testing.T) {
 		})
 	}
 }
+
+// TestTwoMetricsRoutersInOneProcess pins
+// https://git.eeqj.de/sneak/webhooker/issues/227: a second
+// metrics-enabled router in one process used to panic, because the
+// HTTP metrics registered on Prometheus's global default registry.
+// Two routers are built over separate dependency graphs and a third
+// over the first graph again, and each must still serve the HTTP,
+// delivery, Go runtime and process series, and the series counting
+// scrapes of /metrics itself.
+func TestTwoMetricsRoutersInOneProcess(t *testing.T) {
+	t.Parallel()
+
+	first := newTestEnvWithConfig(
+		t, metricsConfig(t, metricsUser, metricsAuthValue),
+	)
+	second := newTestEnvWithConfig(
+		t, metricsConfig(t, metricsUser, metricsAuthValue),
+	)
+	third := &testEnv{
+		router: server.NewRouterForTest(
+			first.log.Get(), first.cfg, first.mw, first.hnd,
+		),
+	}
+
+	for _, env := range []*testEnv{first, second, third} {
+		env.get("/", nil)
+
+		scrape := env.metricsRequest(metricsUser, metricsAuthValue)
+		require.Equal(t, http.StatusOK, scrape.Code)
+
+		for _, series := range []string{
+			"http_request_duration_seconds",
+			"http_response_size_bytes",
+			"http_requests_inflight",
+			"webhooker_events_received_total",
+			"go_goroutines",
+			"process_start_time_seconds",
+			"promhttp_metric_handler_requests_total",
+		} {
+			assert.Contains(t, scrape.Body.String(), series)
+		}
+	}
+}

internal/versionscript/version_script_test.go

@@ -115,8 +115,8 @@ func TestVersion_EnclosingRepositoryIsNotUsed(t *testing.T) {
 	require.Equal(t, unknown, runScript(t, inner, nil))
 }
 
-// The Docker build has no git metadata, so the version arrives as an
+// An explicit VERSION, such as the Dockerfile's build arg, wins over
-// environment override. It wins over anything derivable.
+// anything derivable.
 func TestVersion_EnvironmentOverrideWins(t *testing.T) {
 	t.Parallel()
 
@@ -128,8 +128,8 @@ func TestVersion_EnvironmentOverrideWins(t *testing.T) {
 }
 
 // An empty VERSION is treated as unset rather than stamping an empty
-// string: the Dockerfile's build arg has a non-empty default, but a
+// string: a caller exporting VERSION= must not produce a binary
-// caller exporting VERSION= must not produce a binary reporting "".
+// reporting "".
 func TestVersion_EmptyOverrideFallsBackToGit(t *testing.T) {
 	t.Parallel()
 
@@ -168,8 +168,8 @@ func TestMakefile_BuildComposesVersionAndExtraFlags(t *testing.T) {
 }
 
 // A caller can define VERSION as the empty string -- `make build
-// VERSION=`, or a `--build-arg VERSION=` reaching the Dockerfile's `make
+// VERSION=`, or the Dockerfile's `make build VERSION="$VERSION"` when no
-// build VERSION="$VERSION"`. script/version's own guard does not cover
+// VERSION build arg was given. script/version's own guard does not cover
 // that: the value never passes through the script. Stamping "" would
 // leave the binary reporting no version and the footer on "dev", which
 // is the defect this package exists for.
@@ -231,7 +231,7 @@ func TestDockerfile_BuildsThroughTheMakeTarget(t *testing.T) {
 
 	require.NotContains(t, dockerfile, "go build",
 		"a raw go build bypasses the Makefile's -X flag")
-	require.Contains(t, dockerfile, "ARG VERSION=")
+	require.Contains(t, dockerfile, "ARG VERSION")
 	require.Contains(t, dockerfile,
 		`make build VERSION="$VERSION" GO_LDFLAGS='-extldflags "-static"'`)
 }

script/docker

@@ -2,9 +2,9 @@
 # script/docker: build the Docker image tagged with the project name.
 # The tag comes from script/projectname.
 #
-# .dockerignore excludes .git/, so the builder stage cannot derive the
+# The version script/version resolves here goes in as the VERSION build
-# version itself. It is resolved here, where the checkout is, and passed
+# arg, which takes precedence over what the build would derive from the
-# in as a build arg; without it the image would stamp itself "unknown".
+# .git in its context.
 set -eu
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"

script/version

@@ -7,18 +7,16 @@
 #
 # Order of precedence:
 #
-#   1. $VERSION, if set and non-empty. This is how the value reaches a
+#   1. $VERSION, if set and non-empty: an explicit value, such as the
-#      build that cannot derive it: .dockerignore excludes .git/, so the
+#      Dockerfile's VERSION build arg.
-#      builder stage has no git metadata and the Dockerfile takes the
-#      value as a build arg instead.
 #   2. `git describe --tags --always --dirty` against this checkout. At
 #      a clean tagged commit that is exactly the tag; otherwise it
 #      carries the short SHA, the commit distance when a tag is
 #      reachable, and a -dirty suffix for uncommitted changes.
 #   3. "unknown", for a tree with no git metadata and no $VERSION -- a
-#      source tarball, or `docker build .` with no --build-arg. That
+#      source tarball, or a `docker build` with no .git in its context
-#      case must not fail the build and must not name a tag the tree may
+#      and no VERSION build arg. That case must not fail the build and
-#      not be at, so it names nothing.
+#      must not name a tag the tree may not be at, so it names nothing.
 #
 # The git step insists the enclosing repository is this checkout, not
 # merely some repository above it: an unpacked tarball sitting inside an

templates/error.html

@@ -0,0 +1,15 @@
+{{template "base" .}}
+
+{{define "title"}}{{.StatusText}} - Webhooker{{end}}
+
+{{define "content"}}
+<div class="max-w-4xl mx-auto px-6 py-12">
+    <h1 class="text-2xl font-medium text-gray-900 mb-4">{{.Status}} {{.StatusText}}</h1>
+    <p class="text-gray-600 mb-6">{{.Message}}</p>
+    {{if .User}}
+    <a href="/hooks" class="btn-secondary">Back to webhooks</a>
+    {{else}}
+    <a href="/pages/login" class="btn-primary">Sign in</a>
+    {{end}}
+</div>
+{{end}}

templates/navbar.html

@@ -26,11 +26,15 @@
                 </svg>
                 {{.User.Username}}
             </a>
+            {{/* An error page can be served before a form token is issued,
+                 and a logout without one is refused. */}}
+            {{if .CSRFToken}}
             <form method="POST" action="/pages/logout" class="inline">
                 <input type="hidden" name="csrf_token" value="{{.CSRFToken}}">
                 <button type="submit" class="btn-text">Logout</button>
             </form>
             {{end}}
+            {{end}}
         </div>
     </div>
 
@@ -40,11 +44,13 @@
             {{if .User}}
             <a href="/hooks" class="btn-text w-full text-left">Webhooks</a>
             <a href="/user/{{.User.Username}}" class="btn-text w-full text-left">Profile</a>
+            {{if .CSRFToken}}
             <form method="POST" action="/pages/logout">
                 <input type="hidden" name="csrf_token" value="{{.CSRFToken}}">
                 <button type="submit" class="btn-text w-full text-left">Logout</button>
             </form>
             {{end}}
+            {{end}}
         </div>
     </div>
 </nav>

templates/sources_list.html

@@ -27,10 +27,16 @@
                 </div>
                 <span class="badge-info">Retention: {{.RetentionLabel}}</span>
             </div>
-            <div class="flex gap-6 mt-4 text-sm text-gray-500">
+            <div class="flex flex-wrap gap-6 mt-4 text-sm text-gray-500">
-                <span>{{.EntrypointCount}} entrypoint{{if ne .EntrypointCount 1}}s{{end}}</span>
+                <span>{{.EntrypointCount}} entrypoint{{if ne .EntrypointCount 1}}s{{end}}{{if .InactiveEntrypointCount}}, {{.InactiveEntrypointCount}} inactive{{end}}</span>
-                <span>{{.TargetCount}} target{{if ne .TargetCount 1}}s{{end}}</span>
+                <span>{{.TargetCount}} target{{if ne .TargetCount 1}}s{{end}}{{if .InactiveTargetCount}}, {{.InactiveTargetCount}} inactive{{end}}</span>
-                <span>{{.EventCount}} event{{if ne .EventCount 1}}s{{end}}</span>
+                {{if .EventsUnreadable}}
+                <span class="text-red-600">The event figures could not be read.</span>
+                {{else}}
+                <span>{{.EventCount}} event{{if ne .EventCount 1}}s{{end}} within retention</span>
+                <span>{{with .LastEventAt}}Last event {{.UTC.Format "2006-01-02 15:04:05 UTC"}}{{else}}No events yet{{end}}</span>
+                <span class="{{if .FailedLast24Hours}}font-medium text-red-600{{end}}">{{.FailedLast24Hours}} failed deliver{{if eq .FailedLast24Hours 1}}y{{else}}ies{{end}} in the last 24 hours</span>
+                {{end}}
             </div>
         </a>
         {{end}}