clawbot
bef9986542
All checks were successful
check / check (push) Successful in 3m3s
fx defaults to a 15s stop timeout and the Dockerfile sets no grace override, so Docker SIGKILLed at 10s and the bounded shutdown #130 built — including the log line that tells an operator a component is wedged — was unreachable in the image this repo produces. Sets fx.StopTimeout to 5s, and lowers the HTTP drain to 3s so a full-length drain no longer exhausts the whole sequence budget and skip every later hook, database close included. The Sentry flush, which runs in the same hook and honours no context, is clamped to the remaining stop budget less a 2s tail reserve, so a stalled flush drops Sentry events rather than the database close. Also fixes a latent coin flip in the shared stop-hook waiter, which reported "shutdown timed out" about half the time for a component that drained cleanly against an already-expired context. Independently reviewed three times. The final reviewer derived a stronger invariant than the implementation claims — the server hook's absolute end is bounded at stopTimeout minus the reserve regardless of drain length or of time consumed by preceding hooks — and confirmed the guard's 10ms sweep cannot step over the maximum, since both breakpoints land on its grid. Both Sentry probe arms, the docker stop demo and every mutation were reproduced independently. Known residual, filed separately: the HTTP drain itself is not clamped by the reserve, so slow preceding hooks can still jointly exhaust the budget. Demonstrated with a 2.2s sweeper delay.
76 lines
2.8 KiB
Go
76 lines
2.8 KiB
Go
package main
|
|
|
|
import (
|
|
"testing"
|
|
"time"
|
|
|
|
"github.com/stretchr/testify/require"
|
|
"sneak.berlin/go/webhooker/internal/server"
|
|
)
|
|
|
|
// dockerStopGrace is Docker's default `docker stop` grace period.
|
|
// The Dockerfile sets no STOPSIGNAL or grace override, so this is
|
|
// the deadline the container is actually held to, and the fx stop
|
|
// timeout has to fit inside it with room for signal delivery and
|
|
// process exit.
|
|
const dockerStopGrace = 10 * time.Second
|
|
|
|
// TestNewApp_StopTimeout pins the fx stop timeout. Without the
|
|
// explicit fx.StopTimeout option the app reads fx's 15s
|
|
// DefaultTimeout, which exceeds dockerStopGrace: the container is
|
|
// SIGKILLed before the bound fires and every shutdown hook bounded
|
|
// by it — including the operator-facing timeout log — becomes
|
|
// unreachable in the image this repo produces.
|
|
//
|
|
// fx.New applies options before it executes invokes, so the timeout
|
|
// is set whether or not the graph itself can be constructed here.
|
|
func TestNewApp_StopTimeout(t *testing.T) {
|
|
t.Setenv("DATA_DIR", t.TempDir())
|
|
|
|
got := newApp().StopTimeout()
|
|
|
|
require.Equal(t, stopTimeout, got)
|
|
require.Less(t, got, dockerStopGrace)
|
|
}
|
|
|
|
// tailHeadroom is the slack the fx stop budget must keep beyond the
|
|
// server stop hook. The hooks that run after the server — the
|
|
// delivery engine, the healthcheck, the webhook DB manager and the
|
|
// database close — are microsecond-scale in normal operation, so
|
|
// this is generous for them.
|
|
const tailHeadroom = 2 * time.Second
|
|
|
|
// TestStopTimeout_LeavesHeadroomForTailHooks pins the relationship
|
|
// between the server's stop hook and the fx stop budget. fx bounds
|
|
// the whole stop sequence, and returns without running its
|
|
// remaining hooks once the stop context has expired. If the hook
|
|
// could use the entire budget, every later hook — the database close
|
|
// included — would be skipped in exactly the case where the drain
|
|
// mattered.
|
|
//
|
|
// The hook is not just the HTTP drain: a Sentry flush follows it in
|
|
// the same hook, and sentry.Flush honours no context, so both halves
|
|
// have to be counted. The sweep walks every drain length the hook
|
|
// can produce, since a shorter drain leaves the flush more room and
|
|
// the worst case is not necessarily at either extreme.
|
|
//
|
|
// Shrinking either budget, or unbounding the flush again, must fail
|
|
// here rather than silently recreating a hook that swallows the
|
|
// whole sequence.
|
|
func TestStopTimeout_LeavesHeadroomForTailHooks(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
require.Less(t, server.ShutdownTimeout, stopTimeout)
|
|
|
|
const step = 10 * time.Millisecond
|
|
|
|
for drain := time.Duration(0); drain <= server.ShutdownTimeout; drain += step {
|
|
hook := drain + server.SentryFlushBudget(stopTimeout-drain)
|
|
|
|
require.LessOrEqual(
|
|
t, hook+tailHeadroom, stopTimeout,
|
|
"a %s drain leaves the tail hooks short", drain,
|
|
)
|
|
}
|
|
}
|