Clamp the HTTP drain by the tail-hook reserve (closes #170)

The HTTP drain at shutdown waited up to ShutdownTimeout regardless of how much of the stop budget earlier hooks had used, so a slow archive sweeper or retention reaper could eat the reserve the hooks after the server need, and the database close was skipped. The drain now waits at most the shorter of ShutdownTimeout and what is left of the budget less TailHookReserve, as the Sentry flush already does. The reserve is documented as derived from the two timeouts. Tests cover earlier hooks having spent part of the budget, on a clock that host speed cannot move, and pin that a drain on the full budget gets all of ShutdownTimeout.

Model: opus-5-5

cmd/webhooker/main_test.go

@@ -252,22 +252,40 @@ const tailHeadroom = 2 * time.Second
 // can produce, since a shorter drain leaves the flush more room and
 // the worst case is not necessarily at either extreme.
 //
-// Shrinking either budget, or unbounding the flush again, must fail
-// here rather than silently recreating a hook that swallows the
-// whole sequence.
+// Nor does the hook start on a full budget: the ArchiveSweeper and
+// RetentionReaper hooks run before it, and whatever they spent is
+// gone. The outer sweep walks every amount they can spend. Once they
+// have eaten into the headroom themselves, the hook must spend
+// nothing of what is left. A drain that starts on the full budget
+// must still get all of ShutdownTimeout, so a smaller stopTimeout
+// cannot silently shorten every drain.
+//
+// Shrinking either budget, or unbounding the drain or the flush
+// again, must fail here rather than silently recreating a hook that
+// swallows the whole sequence.
 func TestStopTimeout_LeavesHeadroomForTailHooks(t *testing.T) {
 	t.Parallel()
 
 	require.Less(t, server.ShutdownTimeout, stopTimeout)
+	require.Equal(
+		t, server.ShutdownTimeout, server.DrainBudget(stopTimeout),
+		"a drain that starts on the full stop budget is cut short",
+	)
 
 	const step = 10 * time.Millisecond
 
-	for drain := time.Duration(0); drain <= server.ShutdownTimeout; drain += step {
-		hook := drain + server.SentryFlushBudget(stopTimeout-drain)
+	for spent := time.Duration(0); spent <= stopTimeout; spent += step {
+		remaining := stopTimeout - spent
+		longest := max(server.DrainBudget(remaining), 0)
 
-		require.LessOrEqual(
-			t, hook+tailHeadroom, stopTimeout,
-			"a %s drain leaves the tail hooks short", drain,
-		)
+		for drain := time.Duration(0); drain <= longest; drain += step {
+			hook := drain + server.SentryFlushBudget(remaining-drain)
+
+			require.GreaterOrEqual(
+				t, remaining-hook, min(remaining, tailHeadroom),
+				"a %s drain after %s of earlier hooks leaves "+
+					"the tail hooks short", drain, spent,
+			)
+		}
 	}
 }