Clamp the HTTP drain by the tail-hook reserve (closes #170)

The server's stop hook bounded the drain by ShutdownTimeout alone, so
once the archive sweeper or retention reaper had spent part of the fx
stop budget, a request held open could use up the reserve and fx
skipped every hook after the server, the database close included.
The drain now gets the shorter of ShutdownTimeout and what is left
less TailHookReserve, the clamp the Sentry flush already has.

The headroom test also sweeps the time earlier hooks spent and checks
that a drain on the full budget gets all of ShutdownTimeout. A new
test, run on synctest's clock, holds a request open over net.Pipe
against a stop context with only the reserve left. The README and the
reserve's comment say how the reserve is derived and what a slow
sweeper now costs.

Model: opus-5-5

cmd/webhooker/main.go

@@ -38,17 +38,19 @@ import (
 // hook that used the whole budget would exhaust it at that instant,
 // and fx would skip every hook after the server — the delivery
 // engine, the healthcheck, the webhook DB manager and the database
-// close. That hook is the 3s HTTP drain plus the Sentry flush that
-// follows it in the same hook, so the flush is clamped to the stop
+// close. That hook is the HTTP drain plus the Sentry flush that
+// follows it in the same hook, and each is clamped to the stop
 // context's remaining time less server.TailHookReserve rather than
-// running for its own fixed 2s; the reserve is what the tail hooks
-// live on, and they are microsecond-scale in normal operation.
+// running for its own fixed 3s and 2s; the reserve is what the tail
+// hooks live on, and they are microsecond-scale in normal operation.
 // TestStopTimeout_LeavesHeadroomForTailHooks pins the arithmetic
-// across every drain length.
+// across every drain length and every amount of budget the hooks
+// before the server may already have spent.
 //
 // This does not make the database close unconditional: the
-// ArchiveSweeper and RetentionReaper hooks run before the server
-// and can still consume the whole budget on their own.
+// ArchiveSweeper and RetentionReaper hooks run before the server.
+// What they spend comes out of the drain first, but past 3s it comes
+// out of the reserve, and they can consume the whole budget.
 const stopTimeout = 5 * time.Second
 
 // exitUsage is the status for a command line this binary cannot make

cmd/webhooker/main_test.go

@@ -252,22 +252,40 @@ const tailHeadroom = 2 * time.Second
 // can produce, since a shorter drain leaves the flush more room and
 // the worst case is not necessarily at either extreme.
 //
-// Shrinking either budget, or unbounding the flush again, must fail
-// here rather than silently recreating a hook that swallows the
-// whole sequence.
+// Nor does the hook start on a full budget: the ArchiveSweeper and
+// RetentionReaper hooks run before it, and whatever they spent is
+// gone. The outer sweep walks every amount they can spend. Once they
+// have eaten into the headroom themselves, the hook must spend
+// nothing of what is left. A drain that starts on the full budget
+// must still get all of ShutdownTimeout, so a smaller stopTimeout
+// cannot silently shorten every drain.
+//
+// Shrinking either budget, or unbounding the drain or the flush
+// again, must fail here rather than silently recreating a hook that
+// swallows the whole sequence.
 func TestStopTimeout_LeavesHeadroomForTailHooks(t *testing.T) {
 	t.Parallel()
 
 	require.Less(t, server.ShutdownTimeout, stopTimeout)
+	require.Equal(
+		t, server.ShutdownTimeout, server.DrainBudget(stopTimeout),
+		"a drain that starts on the full stop budget is cut short",
+	)
 
 	const step = 10 * time.Millisecond
 
-	for drain := time.Duration(0); drain <= server.ShutdownTimeout; drain += step {
-		hook := drain + server.SentryFlushBudget(stopTimeout-drain)
+	for spent := time.Duration(0); spent <= stopTimeout; spent += step {
+		remaining := stopTimeout - spent
+		longest := max(server.DrainBudget(remaining), 0)
 
-		require.LessOrEqual(
-			t, hook+tailHeadroom, stopTimeout,
-			"a %s drain leaves the tail hooks short", drain,
-		)
+		for drain := time.Duration(0); drain <= longest; drain += step {
+			hook := drain + server.SentryFlushBudget(remaining-drain)
+
+			require.GreaterOrEqual(
+				t, remaining-hook, min(remaining, tailHeadroom),
+				"a %s drain after %s of earlier hooks leaves "+
+					"the tail hooks short", drain, spent,
+			)
+		}
 	}
 }