Set fx.StopTimeout inside the container stop grace (closes #134)

fx defaults to a 15s stop timeout and the Dockerfile sets no grace
override, so Docker SIGKILLed at 10s and the bounded shutdown #130 built
— including the log line that tells an operator a component is wedged —
was unreachable in the image this repo produces.

Sets fx.StopTimeout to 5s, and lowers the HTTP drain to 3s so a
full-length drain no longer exhausts the whole sequence budget and skip
every later hook, database close included. The Sentry flush, which runs
in the same hook and honours no context, is clamped to the remaining
stop budget less a 2s tail reserve, so a stalled flush drops Sentry
events rather than the database close.

Also fixes a latent coin flip in the shared stop-hook waiter, which
reported "shutdown timed out" about half the time for a component that
drained cleanly against an already-expired context.

Independently reviewed three times. The final reviewer derived a
stronger invariant than the implementation claims — the server hook's
absolute end is bounded at stopTimeout minus the reserve regardless of
drain length or of time consumed by preceding hooks — and confirmed the
guard's 10ms sweep cannot step over the maximum, since both breakpoints
land on its grid. Both Sentry probe arms, the docker stop demo and every
mutation were reproduced independently.

Known residual, filed separately: the HTTP drain itself is not clamped
by the reserve, so slow preceding hooks can still jointly exhaust the
budget. Demonstrated with a 2.2s sweeper delay.

cmd/webhooker/main.go

@@ -2,6 +2,8 @@
 package main
 
 import (
+	"time"
+
 	"go.uber.org/fx"
 	"sneak.berlin/go/webhooker/internal/config"
 	"sneak.berlin/go/webhooker/internal/database"
@@ -15,6 +17,33 @@ import (
 	"sneak.berlin/go/webhooker/internal/session"
 )
 
+// stopTimeout bounds the whole fx stop sequence, not each hook.
+//
+// fx defaults to 15s, which is longer than Docker's 10s default
+// stop grace: the container would be SIGKILLed before the bound
+// could fire, so nothing bounded by it would ever be observed.
+// 5s leaves headroom inside that grace for signal delivery and
+// process exit; the observed wedge case already exits at ~5.3s,
+// so a larger bound would trade a rare skipped database close for
+// a more common hard kill.
+//
+// The server's stop hook must fit inside it with room to spare: a
+// hook that used the whole budget would exhaust it at that instant,
+// and fx would skip every hook after the server — the delivery
+// engine, the healthcheck, the webhook DB manager and the database
+// close. That hook is the 3s HTTP drain plus the Sentry flush that
+// follows it in the same hook, so the flush is clamped to the stop
+// context's remaining time less server.TailHookReserve rather than
+// running for its own fixed 2s; the reserve is what the tail hooks
+// live on, and they are microsecond-scale in normal operation.
+// TestStopTimeout_LeavesHeadroomForTailHooks pins the arithmetic
+// across every drain length.
+//
+// This does not make the database close unconditional: the
+// ArchiveSweeper and RetentionReaper hooks run before the server
+// and can still consume the whole budget on their own.
+const stopTimeout = 5 * time.Second
+
 // Build-time variables set via -ldflags.
 //
 //nolint:gochecknoglobals // Build-time variables injected by the linker.
@@ -27,7 +56,14 @@ func main() {
 	globals.Appname = appname
 	globals.Version = version
 
-	fx.New(
+	newApp().Run()
+}
+
+// newApp builds the application graph. It is separate from main so
+// a test can assert the options it carries.
+func newApp() *fx.App {
+	return fx.New(
+		fx.StopTimeout(stopTimeout),
 		fx.Provide(
 			globals.New,
 			logger.New,
@@ -60,5 +96,5 @@ func main() {
 			) {
 			},
 		),
-	).Run()
+	)
 }

cmd/webhooker/main_test.go

@@ -0,0 +1,75 @@
+package main
+
+import (
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/require"
+	"sneak.berlin/go/webhooker/internal/server"
+)
+
+// dockerStopGrace is Docker's default `docker stop` grace period.
+// The Dockerfile sets no STOPSIGNAL or grace override, so this is
+// the deadline the container is actually held to, and the fx stop
+// timeout has to fit inside it with room for signal delivery and
+// process exit.
+const dockerStopGrace = 10 * time.Second
+
+// TestNewApp_StopTimeout pins the fx stop timeout. Without the
+// explicit fx.StopTimeout option the app reads fx's 15s
+// DefaultTimeout, which exceeds dockerStopGrace: the container is
+// SIGKILLed before the bound fires and every shutdown hook bounded
+// by it — including the operator-facing timeout log — becomes
+// unreachable in the image this repo produces.
+//
+// fx.New applies options before it executes invokes, so the timeout
+// is set whether or not the graph itself can be constructed here.
+func TestNewApp_StopTimeout(t *testing.T) {
+	t.Setenv("DATA_DIR", t.TempDir())
+
+	got := newApp().StopTimeout()
+
+	require.Equal(t, stopTimeout, got)
+	require.Less(t, got, dockerStopGrace)
+}
+
+// tailHeadroom is the slack the fx stop budget must keep beyond the
+// server stop hook. The hooks that run after the server — the
+// delivery engine, the healthcheck, the webhook DB manager and the
+// database close — are microsecond-scale in normal operation, so
+// this is generous for them.
+const tailHeadroom = 2 * time.Second
+
+// TestStopTimeout_LeavesHeadroomForTailHooks pins the relationship
+// between the server's stop hook and the fx stop budget. fx bounds
+// the whole stop sequence, and returns without running its
+// remaining hooks once the stop context has expired. If the hook
+// could use the entire budget, every later hook — the database close
+// included — would be skipped in exactly the case where the drain
+// mattered.
+//
+// The hook is not just the HTTP drain: a Sentry flush follows it in
+// the same hook, and sentry.Flush honours no context, so both halves
+// have to be counted. The sweep walks every drain length the hook
+// can produce, since a shorter drain leaves the flush more room and
+// the worst case is not necessarily at either extreme.
+//
+// Shrinking either budget, or unbounding the flush again, must fail
+// here rather than silently recreating a hook that swallows the
+// whole sequence.
+func TestStopTimeout_LeavesHeadroomForTailHooks(t *testing.T) {
+	t.Parallel()
+
+	require.Less(t, server.ShutdownTimeout, stopTimeout)
+
+	const step = 10 * time.Millisecond
+
+	for drain := time.Duration(0); drain <= server.ShutdownTimeout; drain += step {
+		hook := drain + server.SentryFlushBudget(stopTimeout-drain)
+
+		require.LessOrEqual(
+			t, hook+tailHeadroom, stopTimeout,
+			"a %s drain leaves the tail hooks short", drain,
+		)
+	}
+}