Set fx.StopTimeout inside the container stop grace (closes #134)
All checks were successful
check / check (push) Successful in 3m3s

fx defaults to a 15s stop timeout and the Dockerfile sets no grace
override, so Docker SIGKILLed at 10s and the bounded shutdown #130 built
— including the log line that tells an operator a component is wedged —
was unreachable in the image this repo produces.

Sets fx.StopTimeout to 5s, and lowers the HTTP drain to 3s so a
full-length drain no longer exhausts the whole sequence budget and skip
every later hook, database close included. The Sentry flush, which runs
in the same hook and honours no context, is clamped to the remaining
stop budget less a 2s tail reserve, so a stalled flush drops Sentry
events rather than the database close.

Also fixes a latent coin flip in the shared stop-hook waiter, which
reported "shutdown timed out" about half the time for a component that
drained cleanly against an already-expired context.

Independently reviewed three times. The final reviewer derived a
stronger invariant than the implementation claims — the server hook's
absolute end is bounded at stopTimeout minus the reserve regardless of
drain length or of time consumed by preceding hooks — and confirmed the
guard's 10ms sweep cannot step over the maximum, since both breakpoints
land on its grid. Both Sentry probe arms, the docker stop demo and every
mutation were reproduced independently.

Known residual, filed separately: the HTTP drain itself is not clamped
by the reserve, so slow preceding hooks can still jointly exhaust the
budget. Demonstrated with a 2.2s sweeper delay.
This commit was merged in pull request #159.
This commit is contained in:
2026-08-18 00:12:51 +02:00
parent c3b6623be1
commit bef9986542
8 changed files with 401 additions and 10 deletions

View File

@@ -1145,8 +1145,6 @@ webhooker/
│ │ ├── archive_sweeper.go # Periodic pruning of idle archives
│ │ ├── url_mask.go # Strips credentials from *url.Error
│ │ └── ssrf.go # SSRF prevention (IP validation, safe HTTP transport)
│ ├── lifecycle/
│ │ └── lifecycle.go # Shared fx start/stop hook helpers
│ ├── handlers/
│ │ ├── handlers.go # Base handler struct, JSON helpers, template rendering
│ │ ├── auth.go # Login, logout handlers
@@ -1158,6 +1156,8 @@ webhooker/
│ │ └── webhook.go # Webhook receiver handler
│ ├── healthcheck/
│ │ └── healthcheck.go # Health check service (uptime, version)
│ ├── lifecycle/
│ │ └── lifecycle.go # Shared stop-hook waiter, bounded by the stop context
│ ├── logger/
│ │ └── logger.go # slog setup with TTY detection
│ ├── middleware/
@@ -1316,6 +1316,78 @@ rather than global: **LoginRateLimit** on `/pages/login`,
- GORM soft deletes on every entity that carries `BaseModel`, which is
all of them but `Setting` (data preserved for audit)
### Shutdown
On SIGINT or SIGTERM, fx runs the registered stop hooks in reverse
dependency order under a **5 second budget** (`fx.StopTimeout` in
`cmd/webhooker/main.go`). That budget covers the whole sequence, not
each hook. The order, read off the fx stop-hook log:
1. `ArchiveSweeper`
2. `RetentionReaper`
3. `server` — the HTTP drain, bounded separately by
`server.ShutdownTimeout` (**3 seconds**), then a Sentry flush if
`SENTRY_DSN` is set
4. `delivery.Engine`
5. `healthcheck`
6. `WebhookDBManager`
7. the database close
The two components that can realistically hold the budget run
first: a retention sweep or an archive prune caught mid-tick each
waits on its `WaitGroup` bounded by the stop context, so a wedge
there consumes the 5 seconds before the HTTP server hook is ever
entered. The hooks after the server are microsecond-scale in normal
operation.
The HTTP drain budget is deliberately **shorter** than the sequence
budget. Were the two equal, a drain that used its whole budget would
exhaust the sequence budget at the instant it finished, and every
later hook — the delivery engine, the healthcheck, the webhook DB
manager and the database close — would be skipped in exactly the
case where the drain mattered. 3 seconds leaves 2 seconds
(`server.TailHookReserve`) for the tail, which is far more than the
microseconds it needs.
That reserve belongs to the tail hooks, not to the server hook, and
the Sentry flush is what could take it: it runs after the drain
**inside the same hook**, and `sentry.Flush` takes a bare duration
and honours no context, so an unreachable Sentry endpoint would add
its own timeout on top of a full-length drain and consume the whole
sequence budget by itself. It is therefore clamped to whatever is
left on the stop context minus the reserve, and skipped when that
leaves too little to be worth attempting — so a full-length drain
means Sentry events are dropped rather than the database close being
skipped.
This does not make the database close unconditional: a wedged
`ArchiveSweeper` or `RetentionReaper` still runs first and can
consume the whole budget on its own.
The value is chosen to sit inside the container stop grace period.
Docker's default `docker stop` grace is 10 seconds and the Dockerfile
sets no `STOPSIGNAL` or grace override, so the process must be gone
before that. fx's own default is 15 seconds, which is past the grace:
the container would be SIGKILLed (exit 137) before the bound could
fire, and nothing that depends on it — including the
`shutdown timed out, goroutines still running` error log that tells
an operator a component is wedged — would ever be reached.
Two operational consequences follow from bounding the sequence:
- **A wedged component aborts the rest of the shutdown.** fx checks
the stop context before each remaining hook and returns outright
once it has expired, skipping the hooks it has not reached. If the
first-stopped component consumes the whole budget, the later hooks
never run — **the database close among them**. SQLite is crash-safe,
so this is not corruption, but it is not a clean close either.
- **Lowering the grace below 5 seconds reintroduces the silent
truncation.** `docker stop --time`, Compose's `stop_grace_period`,
or Kubernetes' `terminationGracePeriodSeconds` set under 5 seconds
put SIGKILL back in front of the bound, and the process dies with
no shutdown diagnostics at all. Keep the deployment's grace above
the stop timeout.
### Docker
The Dockerfile uses a three-stage build. Each stage is pinned by