Show each webhook's activity in the webhook list (closes #394)

Each entry in the webhook list now shows when the webhook's last event
arrived, or "No events yet", and how many of its deliveries failed in
the last 24 hours, in red when that is not zero. The entrypoint and
target counts say how many are inactive.

The figures come from the statistics pane's own reads: the event totals
row, which now also gives the event count, shown as events within
retention, instead of counting every stored event, and the pane's query
over the deliveries finished in the last 24 hours. A webhook whose event
database cannot be read says so in its entry rather than showing zeros.

Model: opus-5-5

README.md

@@ -158,19 +158,20 @@ WireServer, which serves an Azure VM its credentials. Because it is a
 public address, listing it in `ALLOWED_EGRESS_CIDRS` reopens it.
 
 That is all the default blocklist covers: the IPv4 private and reserved
-ranges; of IPv6, only loopback (`::1`), unique local addresses
+ranges; of IPv6, only loopback (`::1`), the unspecified address (`::`),
-(`fc00::/7`) and link-local addresses (`fe80::/10`); and certain public
+unique local addresses (`fc00::/7`), link-local addresses (`fe80::/10`),
-addresses. A public address belongs on the default blocklist only if it
+multicast (`ff00::/8`) and documentation space (`2001:db8::/32`); and
-hands credentials, user data or bootstrap material to whatever can reach
+certain public addresses. A public address belongs on the default
-it, without the caller presenting anything. A provider's other public
+blocklist only if it hands credentials, user data or bootstrap material
-addresses are not refused. IBM Cloud, for example, serves its package
+to whatever can reach it, without the caller presenting anything. A
-mirrors, time servers and object storage on `161.26.0.0/16`, and the
+provider's other public addresses are not refused. IBM Cloud, for
-private endpoints of its own cloud services on `166.8.0.0/14`. Neither
+example, serves its package mirrors, time servers and object storage on
-range hands out credentials that way: the token service among those
+`161.26.0.0/16`, and the private endpoints of its own cloud services on
-endpoints issues a token only in exchange for something the caller
+`166.8.0.0/14`. Neither range hands out credentials that way: the token
-presents, such as an API key. Reaching these services can be a
+service among those endpoints issues a token only in exchange for
-legitimate delivery, and every cloud has some, so a partial list would
+something the caller presents, such as an API key. Reaching these
-promise coverage it does not give.
+services can be a legitimate delivery, and every cloud has some, so a
+partial list would promise coverage it does not give.
 
 That default is also inconvenient for the thing webhooker is mostly
 for: taking a public webhook and forwarding it to something on your own
@@ -210,16 +211,16 @@ Two things this setting cannot do:
   the list is always an allowlist; an empty list (the default) means
   every private and reserved range stays refused. Note that
   `0.0.0.0/0` gets you most of the way there anyway, per above.
-- **It cannot open link-local, or a cloud metadata endpoint at a
+- **It cannot open link-local, the unspecified addresses, or a cloud
-  non-public address that discloses credentials or user data.** An
+  metadata endpoint at a non-public address that discloses credentials
-  address is on the list below when it is not a public address and both
+  or user data.** A metadata address is on the list below when it is not
-  of these hold: the provider fixes it, so it cannot collide with
+  a public address and both of these hold: the provider fixes it, so it
-  anything you run; and reaching it hands out credentials, user data or
+  cannot collide with anything you run; and reaching it hands out
-  bootstrap material. Those stay blocked no matter what you list,
+  credentials, user data or bootstrap material. Those stay blocked no
-  including when you list them outright or list a supernet such as
+  matter what you list, including when you list them outright or list a
-  `0.0.0.0/0`, `::/0`, `fd00::/8` or `100.64.0.0/10`. Treat this as best
+  supernet such as `0.0.0.0/0`, `::/0`, `fd00::/8` or `100.64.0.0/10`.
-  effort rather than a guarantee — it is a hand-maintained list and the
+  Treat this as best effort rather than a guarantee — it is a
-  caveat below the table applies:
+  hand-maintained list and the caveat below the table applies:
 
   | Blocked unconditionally | What it is |
   | ----------------------- | ---------- |
@@ -233,14 +234,25 @@ Two things this setting cannot do:
   | `fd00:a9fe:a9fe::1/128` | Linode/Akamai metadata over IPv6 |
   | `100.100.100.200/32` | Alibaba Cloud metadata, inside CGNAT |
   | `192.0.0.192/32` | Oracle Cloud Classic metadata |
+  | `0.0.0.0/32` | IPv4 unspecified address, which reaches this host's loopback on Linux |
+  | `::/128` | IPv6 unspecified address, which reaches this host's loopback on Linux |
   | `::a9fe:a9fe/128` | `169.254.169.254` as an IPv4-compatible IPv6 address |
   | `64:ff9b::a9fe:a9fe/128` | `169.254.169.254` behind the NAT64 well-known prefix |
 
   The IPv4-mapped form `::ffff:169.254.169.254` is covered by the
-  `169.254.0.0/16` entry. Reaching any of these is credential or
+  `169.254.0.0/16` entry. Reaching any of these but the two unspecified
-  user-data theft rather than delivery to an internal service. Every
+  addresses is credential or user-data theft rather than delivery to an
-  entry outside the two link-local blocks is a single address, so
+  internal service. Every entry outside the two link-local blocks is a
-  blocking it costs you nothing else on the network around it.
+  single address, so blocking it costs you nothing else on the network
+  around it.
+
+  The unspecified addresses `0.0.0.0` and `::` hand out nothing
+  themselves, but no host can have either, and on Linux a connection to
+  one reaches this host's own loopback. They are listed so that an
+  allowlist reaches loopback only through an entry that covers a loopback
+  address, such as `127.0.0.0/8`, `::1` or `0.0.0.0/0`, never through one
+  that covers only `0.0.0.0` or `::`; `0.0.0.0/8`, for example, does not
+  open loopback.
 
   The six ULA entries, all inside `fd00::/8`, are why this matters in
   practice: `fd00::/8` is an ordinary block to allowlist for your own
@@ -1768,7 +1780,7 @@ retries) is individually logged for full observability.
 #### EventTotals and TargetTotals
 
 Running counts in each event database, read by the statistics pane at the
-top of the webhook page. `EventTotals` is one row:
+top of the webhook page and by the webhook list. `EventTotals` is one row:
 
 | Field            | Type      | Description |
 | ---------------- | --------- | ----------- |
@@ -1800,6 +1812,14 @@ target. Its failure percentage for a window is the deliveries that became
 `failed` in it out of all that became `delivered` or `failed` in it, and
 a dash when none did.
 
+The webhook list at `/hooks` shows three of the pane's figures for each
+webhook: its events within retention and its last event, both from
+`EventTotals`, and its deliveries that failed in the last 24 hours,
+counted with the pane's query. It opens each webhook's event database once
+(the handle stays open) and runs those two reads there, so its cost grows
+with the number of webhooks and, for each, with the deliveries that
+finished in the last 24 hours, never with the events stored.
+
 #### Event-tier indexes
 
 These indexes on the per-webhook event databases are declared in the model
@@ -1807,7 +1827,7 @@ tags, so `AutoMigrate` creates them on a fresh database:
 
 | Table              | Columns                     | Serves |
 | ------------------ | --------------------------- | ------ |
-| `deliveries`       | `status`, `deleted_at`, `finished_at`, `target_id` | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status, and the webhook page's statistics, which count each target's deliveries by status and when they finished |
+| `deliveries`       | `status`, `deleted_at`, `finished_at`, `target_id` | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status, and the webhook page's statistics and the webhook list, which count each target's deliveries by status and when they finished |
 | `deliveries`       | `event_id`, `deleted_at`    | The event log, which loads each event's deliveries, and retention, which counts and deletes the deliveries of expired events |
 | `delivery_results` | `delivery_id`, `deleted_at` | The event log, which loads the attempts of a page's deliveries, and retention, which deletes the attempts of expired events |
 | `events`           | `deleted_at`, `created_at`  | The webhook page's statistics, which count recent events |
@@ -3093,7 +3113,8 @@ check, see [The login endpoint](#the-login-endpoint).
   route through a single decision function, so they cannot disagree
   about a destination. An operator can permit specific blocks with
   [`ALLOWED_EGRESS_CIDRS`](#allowing-egress-to-your-own-network); the
-  guard cannot be switched off, and link-local plus a
+  guard cannot be switched off, and link-local, the unspecified
+  addresses `0.0.0.0` and `::`, and a
   [pinned set](#allowing-egress-to-your-own-network) of known cloud
   metadata endpoints — several of which are ULAs outside link-local —
   stay blocked whatever is listed, though listing `0.0.0.0/0` or

internal/config/config.go

@@ -196,12 +196,13 @@ type Config struct {
 	// otherwise refuse. The guard itself is always on: there is no
 	// setting that disables SSRF protection, and delivery's
 	// alwaysBlockedNetworks stays blocked no matter what is listed
-	// here. That set is link-local plus the cloud metadata
+	// here. That set is link-local, the unspecified addresses
-	// endpoints outside it that disclose credentials or user data
+	// 0.0.0.0 and ::, and the cloud metadata endpoints outside
-	// at a provider-fixed, non-public address; it is not
+	// link-local that disclose credentials or user data at a
-	// exhaustive of every cloud's metadata address. See
+	// provider-fixed, non-public address; it is not exhaustive of
-	// alwaysBlockedNetworks for the authoritative list and the
+	// every cloud's metadata address. See
-	// criterion it is built from.
+	// alwaysBlockedNetworks for the authoritative list and why
+	// each entry is on it.
 	AllowedEgressCIDRs []netip.Prefix
 
 	params *ConfigParams

internal/config/config_test.go

@@ -124,6 +124,11 @@ func testEnvironmentConfigSuccess(
 
 	app := fxtest.New(
 		t,
+		// fx's own log is discarded, not sent to t.Logf: a hook still
+		// running after a start or stop timeout would write there after
+		// the test has returned. The same holds for every fxtest.New
+		// below.
+		fx.NopLogger,
 		fx.Provide(
 			globals.New,
 			logger.New,
@@ -272,6 +277,7 @@ func testRetentionSweepIntervalSuccess(
 
 	app := fxtest.New(
 		t,
+		fx.NopLogger,
 		fx.Provide(
 			globals.New,
 			logger.New,
@@ -364,6 +370,7 @@ func testSessionIdleTimeoutSuccess(
 
 	app := fxtest.New(
 		t,
+		fx.NopLogger,
 		fx.Provide(
 			globals.New,
 			logger.New,
@@ -404,6 +411,7 @@ func TestDefaultDataDir(t *testing.T) {
 
 			app := fxtest.New(
 				t,
+				fx.NopLogger,
 				fx.Provide(
 					globals.New,
 					logger.New,
@@ -534,6 +542,7 @@ func testReceiverRateLimitSuccess(
 
 	app := fxtest.New(
 		t,
+		fx.NopLogger,
 		fx.Provide(
 			globals.New,
 			logger.New,
@@ -650,6 +659,7 @@ func testTrustedProxiesSuccess(
 
 	app := fxtest.New(
 		t,
+		fx.NopLogger,
 		fx.Provide(
 			globals.New,
 			logger.New,
@@ -763,6 +773,7 @@ func testAllowedEgressCIDRsSuccess(
 
 	app := fxtest.New(
 		t,
+		fx.NopLogger,
 		fx.Provide(
 			globals.New,
 			logger.New,
@@ -1006,6 +1017,7 @@ func assertMetricsAuthAccepted(t *testing.T, expectAuth bool) {
 
 	app := fxtest.New(
 		t,
+		fx.NopLogger,
 		fx.Provide(globals.New, logger.New, config.New),
 		fx.Populate(&cfg),
 	)

internal/delivery/ssrf.go

@@ -37,8 +37,8 @@ var (
 		"blocked cloud metadata address",
 	)
 	errBlockedMetadata = errors.New(
-		"blocked link-local or cloud instance metadata " +
+		"blocked link-local, cloud instance metadata or " +
-			"address: ALLOWED_EGRESS_CIDRS cannot open it",
+			"unspecified address: ALLOWED_EGRESS_CIDRS cannot open it",
 	)
 	errInvalidScheme = errors.New(
 		"only http and https are allowed",
@@ -72,14 +72,17 @@ var blockedNetworks []*net.IPNet
 var blockedPublicNetworks []*net.IPNet
 
 // alwaysBlockedNetworks are the ranges no configuration can
-// open: the link-local blocks and the cloud instance metadata
+// open, so a supplied CIDR that covers one still leaves it
-// endpoints that live outside them. Reaching one is credential
+// blocked. An entry is here for one of two reasons: it is a
-// or user-data theft rather than delivery to an internal
+// metadata endpoint (the link-local blocks and the cloud
-// service, so a supplied CIDR that covers such an address still
+// instance metadata endpoints that live outside them), or it is
-// leaves it blocked.
+// an unspecified address. Reaching a metadata endpoint is
+// credential or user-data theft rather than delivery to an
+// internal service.
 //
-// Inclusion criterion — an address belongs here only if BOTH
+// Inclusion criterion for metadata endpoints — one belongs here
-// hold, and every entry below satisfies both:
+// only if BOTH hold, and every metadata entry below satisfies
+// both:
 //
 //  1. It is a fixed address assigned by the provider, or a
 //     range reserved by IANA — never one the operator chose.
@@ -90,8 +93,8 @@ var blockedPublicNetworks []*net.IPNet
 //     not cheaply rotated.
 //
 // Both halves are load-bearing, so use them to refuse a
-// candidate and say why. An endpoint disclosing only the
+// metadata candidate and say why. An endpoint disclosing only
-// operator's own inventory (instance id, region, disks, NICs)
+// the operator's own inventory (instance id, region, disks, NICs)
 // fails (2): letting a delivery target reach the operator's own
 // infrastructure is the feature ALLOWED_EGRESS_CIDRS exists to
 // provide. But (2) is not "IAM credentials only" either —
@@ -112,6 +115,15 @@ var blockedPublicNetworks []*net.IPNet
 // This is a criterion, not an enumeration of every metadata
 // address in existence.
 //
+// The unspecified addresses 0.0.0.0 and :: are here for a
+// separate reason: they disclose nothing, but no host can have
+// either, and on Linux a connection to one reaches this host's
+// own loopback. Listing them means an allowlist reaches loopback
+// only through an entry that covers a loopback address
+// (127.0.0.0/8, ::1/128, 0.0.0.0/0), never through one that
+// covers only 0.0.0.0 or :: (0.0.0.0/8, for example). Nothing
+// else lives at either address, so refusing them costs nothing.
+//
 // Every entry is either already in blockedNetworks — this list is
 // what makes it unconditional — or an alternate encoding of
 // 169.254.169.254 that Contains does not match against
@@ -131,23 +143,46 @@ var alwaysBlockedNetworks []*net.IPNet
 //nolint:gochecknoinits // init is the idiomatic way to parse CIDRs once at startup
 func init() {
 	blockedNetworks = mustParseCIDRs([]string{
+		// IPv4 loopback.
 		"127.0.0.0/8",
+		// RFC 1918 private network.
 		"10.0.0.0/8",
+		// RFC 1918 private network.
 		"172.16.0.0/12",
+		// RFC 1918 private network.
 		"192.168.0.0/16",
+		// IPv4 link-local.
 		"169.254.0.0/16",
+		// "This network", holding the IPv4 unspecified address 0.0.0.0.
 		"0.0.0.0/8",
+		// Carrier-grade NAT shared address space.
 		"100.64.0.0/10",
+		// IETF protocol assignments.
 		"192.0.0.0/24",
+		// IPv4 documentation (TEST-NET-1).
 		"192.0.2.0/24",
+		// Benchmarking.
 		"198.18.0.0/15",
+		// IPv4 documentation (TEST-NET-2).
 		"198.51.100.0/24",
+		// IPv4 documentation (TEST-NET-3).
 		"203.0.113.0/24",
+		// IPv4 multicast.
 		"224.0.0.0/4",
+		// Reserved, including the broadcast address.
 		"240.0.0.0/4",
+		// IPv6 loopback.
 		"::1/128",
+		// IPv6 unspecified address.
+		"::/128",
+		// IPv6 unique local addresses.
 		"fc00::/7",
+		// IPv6 link-local.
 		"fe80::/10",
+		// IPv6 multicast.
+		"ff00::/8",
+		// IPv6 documentation.
+		"2001:db8::/32",
 	})
 
 	blockedPublicNetworks = mustParseCIDRs([]string{
@@ -207,6 +242,14 @@ func init() {
 		// allowlist from opening it.
 		"192.0.0.192/32",
 
+		// The unspecified addresses, each of which reaches this
+		// host's loopback on Linux.
+		//
+		// IPv4 unspecified address, inside the blocked 0.0.0.0/8.
+		"0.0.0.0/32",
+		// IPv6 unspecified address.
+		"::/128",
+
 		// 169.254.169.254 as an IPv4-compatible IPv6 address.
 		"::a9fe:a9fe/128",
 		// 169.254.169.254 behind the NAT64 well-known prefix.
@@ -343,8 +386,9 @@ func (g *Guard) allows(ip net.IP) bool {
 // The order is the policy:
 //
 //  1. alwaysBlockedNetworks is refused before the allowlist is
-//     consulted, so no configured CIDR reaches link-local or a
+//     consulted, so no configured CIDR reaches link-local, a
-//     cloud metadata endpoint at a non-public address.
+//     cloud metadata endpoint at a non-public address, or an
+//     unspecified address.
 //  2. The allowlist is consulted next, so a listed private
 //     network, or a listed public address on the default
 //     blocklist, becomes reachable.

internal/delivery/ssrf_allowlist_test.go

@@ -168,12 +168,13 @@ func TestGuardAllowlist_UnlistedPrivateStillRefused(t *testing.T) {
 
 // TestGuardAllowlist_MetadataAlwaysRefused is the load-bearing
 // case: cloud instance metadata endpoints are credential theft
-// rather than delivery to an internal service, so no allowlist
+// rather than delivery to an internal service, and the
-// reaches one. Every guard below names a CIDR that covers its
+// unspecified addresses 0.0.0.0 and :: reach this host's loopback
-// target — including 0.0.0.0/0, ::/0, and the ordinary ULA and
+// on Linux, so no allowlist reaches any of them. Every guard
-// CGNAT blocks an operator would really list — and the address
+// below names a CIDR that covers its target — including
-// must stay refused anyway, on both the validation and the
+// 0.0.0.0/0, ::/0, and the ordinary ULA and CGNAT blocks an
-// delivery path.
+// operator would really list — and the address must stay
+// refused anyway, on both the validation and the delivery path.
 func TestGuardAllowlist_MetadataAlwaysRefused(t *testing.T) {
 	t.Parallel()
 
@@ -219,15 +220,17 @@ type metadataAlwaysRefusedCase struct {
 }
 
 // metadataAlwaysRefusedCases enumerates every unconditionally
-// blocked address together with an allowlist entry that would
+// blocked address (link-local, the cloud metadata endpoints and
-// otherwise reach it. Split by family of address only to stay
+// the unspecified addresses) together with an allowlist entry
-// under the function-length limit.
+// that would otherwise reach it. Split by family of address only
+// to stay under the function-length limit.
 func metadataAlwaysRefusedCases() []metadataAlwaysRefusedCase {
 	cases := linkLocalRefusedCases()
 	cases = append(cases, ulaMetadataRefusedCases()...)
 	cases = append(cases, ipv4MetadataRefusedCases()...)
+	cases = append(cases, encodedMetadataRefusedCases()...)
 
-	return append(cases, encodedMetadataRefusedCases()...)
+	return append(cases, unspecifiedRefusedCases()...)
 }
 
 // linkLocalRefusedCases covers the link-local blocks, including
@@ -367,6 +370,23 @@ func encodedMetadataRefusedCases() []metadataAlwaysRefusedCase {
 	}
 }
 
+// unspecifiedRefusedCases covers the unspecified addresses, each
+// of which reaches this host's loopback on Linux.
+func unspecifiedRefusedCases() []metadataAlwaysRefusedCase {
+	return []metadataAlwaysRefusedCase{
+		{
+			name:   "IPv4 unspecified address under 0.0.0.0/0",
+			allow:  allowAllIPv4,
+			target: "http://0.0.0.0:8080/hook",
+		},
+		{
+			name:   "IPv6 unspecified address under ::/0",
+			allow:  allowAllIPv6,
+			target: "http://[::]:8080/hook",
+		},
+	}
+}
+
 // TestGuardAllowlist_PublicUnaffected asserts the allowlist does
 // not narrow anything: public addresses were reachable before it
 // existed and stay reachable, whether or not a list is set.
@@ -524,6 +544,10 @@ func TestAlwaysBlockedNetworks_PinnedSet(t *testing.T) {
 		// Oracle Cloud Classic metadata, inside the blocked
 		// 192.0.0.0/24.
 		"192.0.0.192/32",
+		// The IPv4 and IPv6 unspecified addresses, each of
+		// which reaches this host's loopback on Linux.
+		"0.0.0.0/32",
+		"::/128",
 		// 169.254.169.254 as an IPv4-compatible IPv6 address.
 		"::a9fe:a9fe/128",
 		// 169.254.169.254 behind the NAT64 well-known prefix.
@@ -556,7 +580,8 @@ func TestDefaultBlocklist_PinnedSet(t *testing.T) {
 		{cidr: "172.16.0.0/12", reopenable: true},
 		{cidr: "192.168.0.0/16", reopenable: true},
 		{cidr: linkLocalIPv4, reopenable: false},
-		{cidr: "0.0.0.0/8", reopenable: true},
+		// Its first address, 0.0.0.0, is in the unconditional set.
+		{cidr: "0.0.0.0/8", reopenable: false},
 		{cidr: "100.64.0.0/10", reopenable: true},
 		{cidr: "192.0.0.0/24", reopenable: true},
 		{cidr: "192.0.2.0/24", reopenable: true},
@@ -566,8 +591,11 @@ func TestDefaultBlocklist_PinnedSet(t *testing.T) {
 		{cidr: "224.0.0.0/4", reopenable: true},
 		{cidr: "240.0.0.0/4", reopenable: true},
 		{cidr: "::1/128", reopenable: true},
+		{cidr: "::/128", reopenable: false},
 		{cidr: "fc00::/7", reopenable: true},
 		{cidr: "fe80::/10", reopenable: false},
+		{cidr: "ff00::/8", reopenable: true},
+		{cidr: "2001:db8::/32", reopenable: true},
 		{cidr: "168.63.129.16/32", public: true, reopenable: true},
 	}
 

internal/delivery/ssrf_test.go

@@ -101,6 +101,42 @@ func TestValidateTargetURL_Blocked(t *testing.T) {
 	}
 }
 
+// TestDefaultGuard_RefusesUnspecifiedMulticastAndDocumentation
+// covers the unspecified addresses and the IPv6 multicast and
+// documentation ranges: with no allowlist set, each is refused
+// both when a target is created and when a delivery dials it.
+func TestDefaultGuard_RefusesUnspecifiedMulticastAndDocumentation(
+	t *testing.T,
+) {
+	t.Parallel()
+
+	guard := delivery.NewTestGuard()
+
+	targets := []string{
+		// The unspecified addresses. On Linux a connection to
+		// either reaches this host's loopback.
+		"http://0.0.0.0:8080/hook",
+		"http://[::]:8080/hook",
+		// IPv6 multicast, all nodes.
+		"http://[ff02::1]/hook",
+		// IPv6 documentation.
+		"http://[2001:db8::1]/hook",
+	}
+
+	for _, target := range targets {
+		t.Run(target, func(t *testing.T) {
+			t.Parallel()
+
+			require.Error(t,
+				guard.ValidateTargetURL(context.Background(), target),
+				"%s must be refused at target creation", target,
+			)
+
+			assertDialRefused(t, guard, target)
+		})
+	}
+}
+
 func TestValidateTargetURL_Allowed(t *testing.T) {
 	t.Parallel()
 

internal/gormlog/firstboot_test.go

@@ -137,6 +137,10 @@ func bootAtDebug(t *testing.T, dataDir string) string {
 
 	app := fxtest.New(
 		t,
+		// fx's own log is discarded, not sent to t.Logf: a hook still
+		// running after a start or stop timeout would write there after
+		// the test has returned.
+		fx.NopLogger,
 		fx.Provide(
 			globals.New,
 			logger.New,

internal/handlers/handlers_test.go

@@ -86,6 +86,10 @@ func newTestApp(
 
 	return fxtest.New(
 		t,
+		// fx's own log is discarded, not sent to t.Logf: a hook still
+		// running after a start or stop timeout would write there after
+		// the test has returned.
+		fx.NopLogger,
 		fx.Provide(
 			globals.New,
 			logger.New,

internal/handlers/source_list_test.go

@@ -0,0 +1,405 @@
+package handlers_test
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"regexp"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"gorm.io/gorm"
+	"gorm.io/gorm/clause"
+	"sneak.berlin/go/webhooker/internal/database"
+	"sneak.berlin/go/webhooker/internal/handlers"
+	"sneak.berlin/go/webhooker/internal/logger"
+	"sneak.berlin/go/webhooker/internal/session"
+)
+
+// failedHighlight is how the list marks a number of failed deliveries
+// that is not zero.
+const failedHighlight = `class="font-medium text-red-600"`
+
+// listWebhook adds a webhook with the given name, owned by the test
+// user.
+func listWebhook(
+	t *testing.T, db *database.Database, name string,
+) *database.Webhook {
+	t.Helper()
+
+	wh := &database.Webhook{UserID: deleteTestUserID, Name: name}
+	require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
+
+	return wh
+}
+
+// addEntrypoints adds the given number of entrypoints, all active or
+// all inactive, to a webhook and returns their paths.
+func addEntrypoints(
+	t *testing.T, db *database.Database, webhookID string,
+	count int, active bool,
+) []string {
+	t.Helper()
+
+	paths := make([]string, count)
+	for i := range paths {
+		paths[i] = statsEntrypoint(t, db, webhookID, active)
+	}
+
+	return paths
+}
+
+// addTargets adds the given number of targets, all active or all
+// inactive, to a webhook and returns them.
+func addTargets(
+	t *testing.T, db *database.Database, webhookID string,
+	count int, active bool,
+) []*database.Target {
+	t.Helper()
+
+	targets := make([]*database.Target, count)
+	for i := range targets {
+		targets[i] = seedTarget(t, db, webhookID, database.TargetTypeLog)
+		require.NoError(t, db.DB().Model(targets[i]).
+			Update("active", active).Error)
+	}
+
+	return targets
+}
+
+// renderWebhookList runs the real webhook list handler as the test user
+// and returns the rendered page.
+func renderWebhookList(
+	t *testing.T, h *handlers.Handlers, sess *session.Session,
+) string {
+	t.Helper()
+
+	cookies := authenticatedCookies(
+		t, sess, deleteTestUserID, deleteTestUsername,
+	)
+
+	w := httptest.NewRecorder()
+	h.HandleSourceList().ServeHTTP(
+		w, getRequest(t, "/hooks", cookies, nil),
+	)
+	require.Equal(t, http.StatusOK, w.Code)
+
+	return w.Body.String()
+}
+
+// listCard returns one webhook's entry in a rendered webhook list, its
+// markup as rendered and its text with the markup taken out and each
+// run of space made one space.
+func listCard(t *testing.T, page, webhookID string) (string, string) {
+	t.Helper()
+
+	_, card, found := strings.Cut(page, `href="/hook/`+webhookID+`"`)
+	require.True(t, found, "the list has no entry for %s", webhookID)
+
+	card, _, _ = strings.Cut(card, "</a>")
+	text := regexp.MustCompile(`<[^>]*>`).ReplaceAllString(card, " ")
+
+	return card, strings.Join(strings.Fields(text), " ")
+}
+
+// receiveEvents posts the given number of events to an entrypoint
+// through the real receiver, and returns the webhook's event database
+// and its events, oldest first.
+func receiveEvents(
+	t *testing.T,
+	h *handlers.Handlers,
+	dbMgr *database.WebhookDBManager,
+	webhookID, path string,
+	count int,
+) (*gorm.DB, []database.Event) {
+	t.Helper()
+
+	router := receiverRouter(h)
+
+	for range count {
+		require.Equal(t, http.StatusOK, postReceiver(t, router, path))
+	}
+
+	webhookDB, err := dbMgr.GetDB(webhookID)
+	require.NoError(t, err)
+
+	events := listEvents(t, webhookDB)
+	require.Len(t, events, count)
+
+	return webhookDB, events
+}
+
+// seedFailingWebhook adds a webhook with six entrypoints, two of them
+// inactive, and seven targets, five of them inactive. Four events reach
+// its two active targets, each event arriving at a different time.
+// Three deliveries failed in the last 24 hours, two to the first target
+// and one to the second, one failed 30 hours ago, and one was
+// delivered. It returns the webhook and its newest event.
+func seedFailingWebhook(
+	t *testing.T,
+	h *handlers.Handlers,
+	db *database.Database,
+	dbMgr *database.WebhookDBManager,
+) (*database.Webhook, database.Event) {
+	t.Helper()
+
+	wh := listWebhook(t, db, "failing")
+	paths := addEntrypoints(t, db, wh.ID, 4, true)
+	addEntrypoints(t, db, wh.ID, 2, false)
+
+	active := addTargets(t, db, wh.ID, 2, true)
+	first, second := active[0], active[1]
+
+	addTargets(t, db, wh.ID, 5, false)
+
+	webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, paths[0], 4)
+	now := time.Now()
+
+	statsAge(t, webhookDB, events[0].ID, now.Add(-31*time.Hour))
+	statsAge(t, webhookDB, events[1].ID, now.Add(-2*time.Hour))
+	statsAge(t, webhookDB, events[2].ID, now.Add(-10*time.Minute))
+
+	statsFinish(t, webhookDB,
+		statsDelivery(t, webhookDB, events[0].ID, first.ID),
+		database.DeliveryStatusFailed, now.Add(-30*time.Hour))
+	statsFinish(t, webhookDB,
+		statsDelivery(t, webhookDB, events[1].ID, first.ID),
+		database.DeliveryStatusFailed, now.Add(-time.Hour))
+	statsFinish(t, webhookDB,
+		statsDelivery(t, webhookDB, events[2].ID, first.ID),
+		database.DeliveryStatusFailed, now.Add(-time.Minute))
+	statsFinish(t, webhookDB,
+		statsDelivery(t, webhookDB, events[2].ID, second.ID),
+		database.DeliveryStatusFailed, now.Add(-time.Minute))
+	statsFinish(t, webhookDB,
+		statsDelivery(t, webhookDB, events[3].ID, second.ID),
+		database.DeliveryStatusDelivered, now.Add(-time.Minute))
+
+	return wh, events[3]
+}
+
+// seedHealthyWebhook adds a webhook with four entrypoints and two
+// targets, all active, and three events, each delivered to both
+// targets. It returns the webhook and its newest event.
+func seedHealthyWebhook(
+	t *testing.T,
+	h *handlers.Handlers,
+	db *database.Database,
+	dbMgr *database.WebhookDBManager,
+) (*database.Webhook, database.Event) {
+	t.Helper()
+
+	wh := listWebhook(t, db, "healthy")
+	paths := addEntrypoints(t, db, wh.ID, 4, true)
+	targets := addTargets(t, db, wh.ID, 2, true)
+
+	webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, paths[0], 3)
+
+	for _, ev := range events {
+		for _, target := range targets {
+			statsFinish(t, webhookDB,
+				statsDelivery(t, webhookDB, ev.ID, target.ID),
+				database.DeliveryStatusDelivered, time.Now())
+		}
+	}
+
+	return wh, events[2]
+}
+
+// lastEventText is how the list shows the arrival of an event.
+func lastEventText(ev database.Event) string {
+	return ev.CreatedAt.UTC().Format("2006-01-02 15:04:05 UTC")
+}
+
+// TestSourceList_ShowsActivityOfEachWebhook checks the figures the list
+// shows for a webhook with recent failures, a healthy one, a new one
+// that has received no event, and one without an event database.
+func TestSourceList_ShowsActivityOfEachWebhook(t *testing.T) {
+	t.Parallel()
+
+	var (
+		h     *handlers.Handlers
+		sess  *session.Session
+		db    *database.Database
+		dbMgr *database.WebhookDBManager
+	)
+
+	app := newTestApp(t, &h, &sess, &db, &dbMgr)
+	app.RequireStart()
+
+	t.Cleanup(app.RequireStop)
+
+	failing, failingNewest := seedFailingWebhook(t, h, db, dbMgr)
+	healthy, healthyNewest := seedHealthyWebhook(t, h, db, dbMgr)
+
+	// Creating a webhook creates its event database.
+	fresh := listWebhook(t, db, "fresh")
+	require.NoError(t, dbMgr.CreateDB(fresh.ID))
+	addEntrypoints(t, db, fresh.ID, 2, true)
+	addTargets(t, db, fresh.ID, 3, true)
+
+	quiet := listWebhook(t, db, "quiet")
+	addEntrypoints(t, db, quiet.ID, 2, true)
+	addTargets(t, db, quiet.ID, 3, true)
+
+	page := renderWebhookList(t, h, sess)
+
+	card, text := listCard(t, page, failing.ID)
+	assert.Contains(t, text, "6 entrypoints, 2 inactive")
+	assert.Contains(t, text, "7 targets, 5 inactive")
+	assert.Contains(t, text, "4 events within retention")
+	assert.Contains(t, text, "Last event "+lastEventText(failingNewest))
+	assert.Contains(t, card,
+		failedHighlight+">3 failed deliveries in the last 24 hours<")
+
+	card, text = listCard(t, page, healthy.ID)
+	assert.Contains(t, text, "4 entrypoints")
+	assert.Contains(t, text, "2 targets")
+	assert.Contains(t, text, "3 events within retention")
+	assert.Contains(t, text, "Last event "+lastEventText(healthyNewest))
+	assert.Contains(t, text, "0 failed deliveries in the last 24 hours")
+	assert.NotContains(t, text, "inactive")
+	assert.NotContains(t, card, failedHighlight)
+
+	card, text = listCard(t, page, fresh.ID)
+	assert.Contains(t, text, "2 entrypoints")
+	assert.Contains(t, text, "3 targets")
+	assert.Contains(t, text, "0 events within retention")
+	assert.Contains(t, text, "No events yet")
+	assert.Contains(t, text, "0 failed deliveries in the last 24 hours")
+	assert.NotContains(t, card, failedHighlight)
+
+	card, text = listCard(t, page, quiet.ID)
+	assert.Contains(t, text, "2 entrypoints")
+	assert.Contains(t, text, "3 targets")
+	assert.Contains(t, text, "0 events within retention")
+	assert.Contains(t, text, "No events yet")
+	assert.Contains(t, text, "0 failed deliveries in the last 24 hours")
+	assert.NotContains(t, card, failedHighlight)
+	assert.False(t, dbMgr.DBExists(quiet.ID),
+		"showing the list must not create an event database")
+}
+
+// TestSourceList_CountsOnlyEventsWithinRetention checks that once
+// retention has removed one of a webhook's three events, the list
+// counts the two still stored.
+func TestSourceList_CountsOnlyEventsWithinRetention(t *testing.T) {
+	t.Parallel()
+
+	var (
+		h     *handlers.Handlers
+		sess  *session.Session
+		db    *database.Database
+		dbMgr *database.WebhookDBManager
+		log   *logger.Logger
+	)
+
+	app := newTestApp(t, &h, &sess, &db, &dbMgr, &log)
+	app.RequireStart()
+
+	t.Cleanup(app.RequireStop)
+
+	wh := &database.Webhook{
+		UserID: deleteTestUserID, Name: "pruned", RetentionDays: 14,
+	}
+	require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
+
+	paths := addEntrypoints(t, db, wh.ID, 3, true)
+	addTargets(t, db, wh.ID, 4, true)
+
+	webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, paths[0], 3)
+
+	statsAge(t, webhookDB, events[0].ID, time.Now().Add(-15*24*time.Hour))
+	statsPrune(t, db, dbMgr, log, webhookDB)
+	require.Len(t, listEvents(t, webhookDB), 2)
+
+	_, text := listCard(t, renderWebhookList(t, h, sess), wh.ID)
+	assert.Contains(t, text, "3 entrypoints")
+	assert.Contains(t, text, "4 targets")
+	assert.Contains(t, text, "2 events within retention")
+}
+
+// TestSourceList_LastEventSurvivesPruningEveryEvent checks that once
+// retention has removed every event of a webhook, the list still shows
+// when the last one arrived rather than "No events yet".
+func TestSourceList_LastEventSurvivesPruningEveryEvent(t *testing.T) {
+	t.Parallel()
+
+	var (
+		h     *handlers.Handlers
+		sess  *session.Session
+		db    *database.Database
+		dbMgr *database.WebhookDBManager
+		log   *logger.Logger
+	)
+
+	app := newTestApp(t, &h, &sess, &db, &dbMgr, &log)
+	app.RequireStart()
+
+	t.Cleanup(app.RequireStop)
+
+	wh := &database.Webhook{
+		UserID: deleteTestUserID, Name: "emptied", RetentionDays: 1,
+	}
+	require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
+
+	paths := addEntrypoints(t, db, wh.ID, 2, true)
+	addTargets(t, db, wh.ID, 3, true)
+
+	webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, paths[0], 1)
+
+	statsAge(t, webhookDB, events[0].ID, time.Now().Add(-50*time.Hour))
+	statsPrune(t, db, dbMgr, log, webhookDB)
+	require.Empty(t, listEvents(t, webhookDB))
+
+	_, text := listCard(t, renderWebhookList(t, h, sess), wh.ID)
+	assert.Contains(t, text, "0 events within retention")
+	assert.Contains(t, text, "Last event "+lastEventText(events[0]))
+	assert.NotContains(t, text, "No events yet")
+}
+
+// TestSourceList_UnreadableEventDatabase checks that a webhook whose
+// event database cannot be read says so in its entry instead of
+// showing zeros, and that the rest of the list is still shown.
+func TestSourceList_UnreadableEventDatabase(t *testing.T) {
+	t.Parallel()
+
+	var (
+		h     *handlers.Handlers
+		sess  *session.Session
+		db    *database.Database
+		dbMgr *database.WebhookDBManager
+	)
+
+	app := newTestApp(t, &h, &sess, &db, &dbMgr)
+	app.RequireStart()
+
+	t.Cleanup(app.RequireStop)
+
+	broken := listWebhook(t, db, "broken")
+	addEntrypoints(t, db, broken.ID, 2, true)
+	addTargets(t, db, broken.ID, 3, true)
+
+	brokenDB, err := dbMgr.GetDB(broken.ID)
+	require.NoError(t, err)
+	require.NoError(t,
+		brokenDB.Migrator().DropTable(&database.EventTotals{}))
+
+	quiet := listWebhook(t, db, "quiet")
+	addEntrypoints(t, db, quiet.ID, 2, true)
+	addTargets(t, db, quiet.ID, 3, true)
+
+	page := renderWebhookList(t, h, sess)
+
+	_, text := listCard(t, page, broken.ID)
+	assert.Contains(t, text, "2 entrypoints")
+	assert.Contains(t, text, "3 targets")
+	assert.Contains(t, text, "The event figures could not be read.")
+	assert.NotContains(t, text, "events")
+	assert.NotContains(t, text, "failed")
+
+	_, text = listCard(t, page, quiet.ID)
+	assert.Contains(t, text, "No events yet")
+}

internal/handlers/source_management.go

@@ -3,10 +3,12 @@ package handlers
 import (
 	"encoding/json"
 	"errors"
+	"fmt"
 	"net/http"
 	"slices"
 	"strconv"
 	"strings"
+	"time"
 
 	"github.com/go-chi/chi"
 	"github.com/google/uuid"
@@ -20,9 +22,20 @@ import (
 type WebhookListItem struct {
 	database.Webhook
 
-	EntrypointCount int64
+	EntrypointCount         int
-	TargetCount     int64
+	InactiveEntrypointCount int
-	EventCount      int64
+	TargetCount             int
+	InactiveTargetCount     int
+
+	// EventCount is how many events the webhook holds, LastEventAt
+	// when the newest arrived (nil before the first), and
+	// FailedLast24Hours how many of its deliveries failed in the last
+	// 24 hours. When the webhook's event database could not be read,
+	// EventsUnreadable is set and these three are not known.
+	EventCount        int64
+	LastEventAt       *time.Time
+	FailedLast24Hours int64
+	EventsUnreadable  bool
 }
 
 // errMissingURL signals that a required URL was not provided.
@@ -154,7 +167,12 @@ func (h *Handlers) HandleSourceList() http.HandlerFunc {
 			return
 		}
 
-		items := h.buildWebhookListItems(webhooks)
+		items, err := h.buildWebhookListItems(webhooks)
+		if err != nil {
+			h.serverError(w, r, "failed to list webhooks", err)
+
+			return
+		}
 
 		data := map[string]any{
 			"Webhooks": items,
@@ -164,36 +182,115 @@ func (h *Handlers) HandleSourceList() http.HandlerFunc {
 	}
 }
 
-// buildWebhookListItems builds list items with counts.
+// buildWebhookListItems builds the list's entry for each webhook. It
+// fails when the main database cannot be read. A webhook whose event
+// database cannot be read is marked on its own entry, and the error is
+// logged.
 func (h *Handlers) buildWebhookListItems(
 	webhooks []database.Webhook,
-) []WebhookListItem {
+) ([]WebhookListItem, error) {
 	items := make([]WebhookListItem, len(webhooks))
+	since := time.Now().Add(-longWindow)
 
 	for i := range webhooks {
-		items[i].Webhook = webhooks[i]
+		item := &items[i]
+		item.Webhook = webhooks[i]
 
-		h.db.DB().Model(&database.Entrypoint{}).Where(
+		var err error
-			"webhook_id = ?", webhooks[i].ID,
-		).Count(&items[i].EntrypointCount)
 
-		h.db.DB().Model(&database.Target{}).Where(
+		item.EntrypointCount, item.InactiveEntrypointCount, err =
-			"webhook_id = ?", webhooks[i].ID,
+			h.countWithInactive(&database.Entrypoint{}, item.ID)
-		).Count(&items[i].TargetCount)
+		if err != nil {
+			return nil, err
+		}
 
-		if h.dbMgr.DBExists(webhooks[i].ID) {
+		item.TargetCount, item.InactiveTargetCount, err =
-			webhookDB, err := h.dbMgr.GetDB(
+			h.countWithInactive(&database.Target{}, item.ID)
-				webhooks[i].ID,
+		if err != nil {
+			return nil, err
+		}
+
+		// Opening an event database that does not exist would create
+		// it, and it would hold nothing to count.
+		if !h.dbMgr.DBExists(item.ID) {
+			continue
+		}
+
+		err = h.readListEventFigures(item, since)
+		if err != nil {
+			h.log.Error(
+				"failed to read webhook list figures",
+				"webhook_id", item.ID,
+				"error", err,
 			)
-			if err == nil {
+
-				webhookDB.Model(
+			item.EventsUnreadable = true
-					&database.Event{},
-				).Count(&items[i].EventCount)
-			}
 		}
 	}
 
-	return items
+	return items, nil
+}
+
+// countWithInactive returns how many entrypoints or targets, as model
+// says, a webhook has, and how many of them are inactive.
+func (h *Handlers) countWithInactive(
+	model any, webhookID string,
+) (int, int, error) {
+	var active []bool
+
+	err := h.db.DB().Model(model).
+		Where("webhook_id = ?", webhookID).
+		Pluck("active", &active).Error
+	if err != nil {
+		return 0, 0, fmt.Errorf(
+			"reading active flags of webhook %s: %w", webhookID, err,
+		)
+	}
+
+	inactive := 0
+
+	for _, a := range active {
+		if !a {
+			inactive++
+		}
+	}
+
+	return len(active), inactive, nil
+}
+
+// readListEventFigures fills in the figures the list shows from the
+// webhook's event database, with the statistics pane's own queries:
+// the event count and last arrival from the event totals row, and the
+// deliveries that failed since the given time from the deliveries'
+// status index.
+func (h *Handlers) readListEventFigures(
+	item *WebhookListItem, since time.Time,
+) error {
+	webhookDB, err := h.dbMgr.GetDB(item.ID)
+	if err != nil {
+		return err
+	}
+
+	var totals database.EventTotals
+
+	err = webhookDB.Take(&totals).Error
+	if err != nil {
+		return fmt.Errorf("reading event totals: %w", err)
+	}
+
+	item.EventCount = totals.Events - totals.EventsRemoved
+	item.LastEventAt = totals.LastEventAt
+
+	byTarget, err := finishedByTarget(webhookDB, since)
+	if err != nil {
+		return err
+	}
+
+	for _, f := range byTarget {
+		item.FailedLast24Hours += f.Failed
+	}
+
+	return nil
 }
 
 // HandleSourceCreate shows the form to create a new webhook.
@@ -1560,10 +1657,11 @@ func (h *Handlers) validateTargetURL(
 		msg := "Invalid target URL: " + err.Error()
 
 		// Only a private or reserved address's refusal says how
-		// to allow it. Metadata refusals never do: link-local and
+		// to allow it. Other refusals never do: link-local, the
-		// the other unconditional metadata addresses cannot be
+		// unspecified addresses and the unconditional metadata
-		// opened, and the default blocklist's public addresses,
+		// addresses cannot be opened, and the default
-		// which listing does open, hand out credentials.
+		// blocklist's public addresses, which listing does open,
+		// hand out credentials.
 		if errors.Is(err, delivery.ErrBlockedPrivateOrReservedIP) {
 			msg += ". Private and reserved addresses are refused " +
 				"by default; the server's ALLOWED_EGRESS_CIDRS " +

internal/resetpw/resetpw_test.go

@@ -152,6 +152,10 @@ func newServerApp(
 
 	app := fxtest.New(
 		t,
+		// fx's own log is discarded, not sent to t.Logf: a hook still
+		// running after a start or stop timeout would write there after
+		// the test has returned.
+		fx.NopLogger,
 		fx.Provide(
 			globals.New,
 			logger.New,

internal/server/routes_test.go

@@ -104,6 +104,10 @@ func newTestEnvWithConfig(
 
 	app := fxtest.New(
 		t,
+		// fx's own log is discarded, not sent to t.Logf: a hook still
+		// running after a start or stop timeout would write there after
+		// the test has returned.
+		fx.NopLogger,
 		fx.Provide(
 			globals.New,
 			logger.New,

templates/sources_list.html

@@ -27,10 +27,16 @@
                 </div>
                 <span class="badge-info">Retention: {{.RetentionLabel}}</span>
             </div>
-            <div class="flex gap-6 mt-4 text-sm text-gray-500">
+            <div class="flex flex-wrap gap-6 mt-4 text-sm text-gray-500">
-                <span>{{.EntrypointCount}} entrypoint{{if ne .EntrypointCount 1}}s{{end}}</span>
+                <span>{{.EntrypointCount}} entrypoint{{if ne .EntrypointCount 1}}s{{end}}{{if .InactiveEntrypointCount}}, {{.InactiveEntrypointCount}} inactive{{end}}</span>
-                <span>{{.TargetCount}} target{{if ne .TargetCount 1}}s{{end}}</span>
+                <span>{{.TargetCount}} target{{if ne .TargetCount 1}}s{{end}}{{if .InactiveTargetCount}}, {{.InactiveTargetCount}} inactive{{end}}</span>
-                <span>{{.EventCount}} event{{if ne .EventCount 1}}s{{end}}</span>
+                {{if .EventsUnreadable}}
+                <span class="text-red-600">The event figures could not be read.</span>
+                {{else}}
+                <span>{{.EventCount}} event{{if ne .EventCount 1}}s{{end}} within retention</span>
+                <span>{{with .LastEventAt}}Last event {{.UTC.Format "2006-01-02 15:04:05 UTC"}}{{else}}No events yet{{end}}</span>
+                <span class="{{if .FailedLast24Hours}}font-medium text-red-600{{end}}">{{.FailedLast24Hours}} failed deliver{{if eq .FailedLast24Hours 1}}y{{else}}ies{{end}} in the last 24 hours</span>
+                {{end}}
             </div>
         </a>
         {{end}}