Set fx.StopTimeout inside the container stop grace (closes #134)
All checks were successful
check / check (push) Successful in 3m16s

fx defaults the stop timeout to 15s and the Dockerfile sets no
STOPSIGNAL or grace override, so Docker's 10s default SIGKILLs the
process five seconds before the bound can fire. Everything gated on
it — including the "shutdown timed out, goroutines still running"
error log that tells an operator a component is wedged — was
unreachable in the image this repo produces.

Set fx.StopTimeout to 5s: inside the grace with headroom for signal
delivery and process exit, and equal to the HTTP server's own drain
budget so the first hook can spend its whole budget without the
sequence bound truncating it. The option set moves into newApp() so
a test can read (*fx.App).StopTimeout() back and pin it against
drift; dropping the option makes that test report fx's 15s default.

Also fix a latent coin flip in the shared stop-hook waiter. It
selected on the drained channel against ctx.Done() with no
preamble, and select picks uniformly among ready cases, so a
component that drained against an already-expired context reported
a timeout about half the time. Not reachable through fx, which
re-checks ctx.Err() before each hook, but the helper is shared and
a direct caller can reach it. waitDone now settles the drained case
in a non-blocking preamble first; the test drives it over 1000
passes, so a restored coin flip cannot pass by luck.

README records the timeout and its relationship to the container
stop grace: that lowering the grace below it puts SIGKILL back in
front of the bound, and that an expired stop context makes fx skip
its remaining hooks, so a wedge in the first-stopped component
means the database close never runs. Adds the missing
internal/lifecycle/ entry to the Package Layout tree.
This commit is contained in:
2026-08-17 20:49:03 +00:00
parent 2ee720a9af
commit c0c13ec80c
6 changed files with 185 additions and 2 deletions

View File

@@ -1040,6 +1040,8 @@ webhooker/
│ │ └── webhook.go # Webhook receiver handler
│ ├── healthcheck/
│ │ └── healthcheck.go # Health check service (uptime, version)
│ ├── lifecycle/
│ │ └── lifecycle.go # Shared stop-hook waiter, bounded by the stop context
│ ├── logger/
│ │ └── logger.go # slog setup with TTY detection
│ ├── middleware/
@@ -1165,6 +1167,39 @@ downstream at form-parse time.
- Container runs as non-root user (UID 1000)
- GORM soft deletes on all entities (data preserved for audit)
### Shutdown
On SIGINT or SIGTERM, fx runs the registered stop hooks in reverse
dependency order under a **5 second budget** (`fx.StopTimeout` in
`cmd/webhooker/main.go`). That budget covers the whole sequence, not
each hook: the HTTP server drains first with its own 5 second limit,
then the delivery engine, the reapers, and finally the database
close.
The value is chosen to sit inside the container stop grace period.
Docker's default `docker stop` grace is 10 seconds and the Dockerfile
sets no `STOPSIGNAL` or grace override, so the process must be gone
before that. fx's own default is 15 seconds, which is past the grace:
the container would be SIGKILLed (exit 137) before the bound could
fire, and nothing that depends on it — including the
`shutdown timed out, goroutines still running` error log that tells
an operator a component is wedged — would ever be reached.
Two operational consequences follow from bounding the sequence:
- **A wedged component aborts the rest of the shutdown.** fx checks
the stop context before each remaining hook and returns outright
once it has expired, skipping the hooks it has not reached. If the
first-stopped component consumes the whole budget, the later hooks
never run — **the database close among them**. SQLite is crash-safe,
so this is not corruption, but it is not a clean close either.
- **Lowering the grace below 5 seconds reintroduces the silent
truncation.** `docker stop --time`, Compose's `stop_grace_period`,
or Kubernetes' `terminationGracePeriodSeconds` set under 5 seconds
put SIGKILL back in front of the bound, and the process dies with
no shutdown diagnostics at all. Keep the deployment's grace above
the stop timeout.
### Docker
The Dockerfile uses a multi-stage build: