Bound each shutdown stage by the stop deadline (closes #100)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
Shutdown passes its context to every stage: the streamer, the four handlers, the WHOIS fetcher, the database maintainer and the HTTP server each stop waiting when it ends. A handler then skips its last write and logs how many updates it did not write, and the HTTP server closes the connections still open. The database is checkpointed and closed last, and left open once the deadline has passed, since a write may still hold it. The stop hook gives Shutdown 8 seconds and fx 9, inside the 10 seconds upaas and docker stop allow, so a stop signal exits 0. Model: opus-5-5
This commit was merged in pull request #130.
This commit is contained in:
@@ -202,7 +202,11 @@ routewatch/
|
||||
batches for performance
|
||||
- **Backpressure**: Probabilistic message dropping when queues exceed 50%
|
||||
capacity
|
||||
- **Graceful Shutdown**: 60-second timeout, flushes all pending batches
|
||||
- **Graceful Shutdown**: on SIGTERM or SIGINT the streamer stops, the handlers
|
||||
write the batches they hold, the HTTP server stops and the database is
|
||||
checkpointed and closed. The daemon gives this 8 seconds and exits with status
|
||||
0; "Running under upaas" says what a stop loses, logs and leaves undone when
|
||||
the 8 seconds run out.
|
||||
- **Reconnection**: Any end of the stream other than shutdown, including RIS
|
||||
Live ending the response normally, marks the feed disconnected and reconnects
|
||||
with exponential backoff (5s-320s). The backoff resets after a stream that
|
||||
@@ -369,6 +373,24 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app needs:
|
||||
`/.well-known/healthcheck.json` on the container port. upaas reads the
|
||||
container's health 60 seconds after a deploy and fails the deploy unless it is
|
||||
`healthy`.
|
||||
- Stop grace: the container needs 9 seconds between SIGTERM and SIGKILL. upaas
|
||||
gives 10 (`stopTimeoutSeconds` in its `internal/docker/client.go`), as
|
||||
`docker stop` does by default. The daemon allows 8 seconds for the handlers to
|
||||
write what they hold and for the requests in progress to finish, then stops
|
||||
the HTTP server, closes the database and exits with status 0. A longer grace
|
||||
changes nothing. What is not written after 8 seconds is lost, and the streamer
|
||||
and each handler log how much. When the prefix handler has fallen behind the
|
||||
feed, a stop loses far more: its batch is full at 250,000 updates and the
|
||||
streamer's worker for it waits for room, so the streamer's stop takes the
|
||||
whole 8 seconds and every handler then skips its last write. Lost then are the
|
||||
messages left on the four handlers' queues, what the AS, peer and peering
|
||||
handlers hold, the prefix handler's batch, and the write it has in progress,
|
||||
up to another 250,000 updates, which take about 20 seconds and are cut off by
|
||||
the exit; withdrawals and peer downs are among them. A route withdrawn in them
|
||||
stays in the live route table until its peer announces or withdraws the prefix
|
||||
again or its session goes down. When the 8 seconds run out the database is not
|
||||
closed, and SQLite rolls back a write the exit cut off when the database next
|
||||
opens.
|
||||
|
||||
## Development
|
||||
|
||||
|
||||
Reference in New Issue
Block a user