Drain on SIGTERM instead of dropping everything (K-W5) #132

Merged
jcoffey-dev merged 1 commits from feat/graceful-shutdown into main 2026-11-20 07:25:41 +00:00
Owner

Closes K-W5 (#107).

Before: SIGTERM (a rollout, docker stop) exited at once. A JMAP request in flight failed in the browser, and one that reached the pod as it went hung until its timeout: the error a user sees.

Now (server/src/drain.ts):

  1. Draining: /api/health answers 503 and every response says Connection: close, so load balancers and keep-alive clients move to another replica. Requests are still served, for SHUTDOWN_DELAY (default 3 s).
  2. No new connections, and every live-update stream is ended cleanly: fan-out tabs, the relay and the raw relay. Each tab's EventSource reconnects through the load balancer to a replica that's staying, as it already does when a stream ends (push.ts, 1 s backoff).
  3. In-flight requests get SHUTDOWN_TIMEOUT (default 5 s) to finish; what's left is closed and the process exits. Idle keep-alive connections are closed as they free up, so a finished request doesn't hold the server open. A second signal stops at once.

The defaults fit inside docker stop's 10 s and Kubernetes' 30 s grace. Both are in .env.example.

Tests: five new in drain.test.ts (503 + Connection: close while draining, each stream ended once, a late stream ended at once, an in-flight request finishes, a stuck one is closed at the timeout); typecheck, all 321 server tests, the release-version test and the build pass locally.

In the lab (inbuxa/inbuxa-k8s-lab#36, webmail-restart-test.sh): a rolling restart of the webmail Deployment under three open tabs and a JMAP request every 200 ms, the client in a pod going through the Service:

Requests Failed Streams
released 27.1-rc.10 83 1 (hung 30 s) ended with the pod
this branch, 3 runs 183, 182, 182 0 every one ended cleanly and reconnected
Closes K-W5 (#107). **Before:** SIGTERM (a rollout, `docker stop`) exited at once. A JMAP request in flight failed in the browser, and one that reached the pod as it went hung until its timeout: the error a user sees. **Now** (`server/src/drain.ts`): 1. **Draining:** `/api/health` answers 503 and every response says `Connection: close`, so load balancers and keep-alive clients move to another replica. Requests are still served, for `SHUTDOWN_DELAY` (default 3 s). 2. **No new connections**, and every live-update stream is ended cleanly: fan-out tabs, the relay and the raw relay. Each tab's EventSource reconnects through the load balancer to a replica that's staying, as it already does when a stream ends (`push.ts`, 1 s backoff). 3. **In-flight requests** get `SHUTDOWN_TIMEOUT` (default 5 s) to finish; what's left is closed and the process exits. Idle keep-alive connections are closed as they free up, so a finished request doesn't hold the server open. A second signal stops at once. The defaults fit inside `docker stop`'s 10 s and Kubernetes' 30 s grace. Both are in `.env.example`. **Tests:** five new in `drain.test.ts` (503 + `Connection: close` while draining, each stream ended once, a late stream ended at once, an in-flight request finishes, a stuck one is closed at the timeout); typecheck, all 321 server tests, the release-version test and the build pass locally. **In the lab** (inbuxa/inbuxa-k8s-lab#36, `webmail-restart-test.sh`): a rolling restart of the webmail Deployment under three open tabs and a JMAP request every 200 ms, the client in a pod going through the Service: | | Requests | Failed | Streams | |---|---|---|---| | released 27.1-rc.10 | 83 | **1** (hung 30 s) | ended with the pod | | this branch, 3 runs | 183, 182, 182 | **0** | every one ended cleanly and reconnected |
jcoffey-dev added 1 commit 2026-11-20 07:23:55 +00:00
Drain on SIGTERM instead of dropping everything (K-W5)
ci / node (pull_request) Skipped
ci / version (pull_request) Skipped
ci / docker-build (pull_request) Skipped
ci / publish (pull_request) Skipped
github/ci (branch) GitHub Actions
ci / github (pull_request) Successful in 1m43s
ci / announce (pull_request) Skipped
8388c97089
On SIGTERM the server exited at once: a JMAP request in flight failed in
the browser, and a request reaching the pod as it went hung until it
timed out. Now it drains (server/src/drain.ts):

1. /api/health answers 503 and every response says Connection: close,
   so load balancers and keep-alive clients move to another replica;
   requests are still served, for SHUTDOWN_DELAY (3 s).
2. No new connections, and every live-update stream (fan-out, relay and
   raw relay) is ended cleanly, so each tab's EventSource reconnects to a
   replica that's staying.
3. Requests in flight get SHUTDOWN_TIMEOUT (5 s) to finish; then what's
   left is closed and the process exits. A second signal stops at once.

Tested in the lab (inbuxa-k8s-lab webmail-restart-test.sh): a rolling
restart under three open tabs and a JMAP request every 200 ms. The
released webmail had a request hang for 30 s; with this, 182-183
requests and none failed, three runs out of three, every stream ending
cleanly and reconnecting.
jcoffey-dev merged commit ba96d33201 into main 2026-11-20 07:25:41 +00:00
jcoffey-dev deleted branch feat/graceful-shutdown 2026-11-20 07:25:41 +00:00
Sign in to join this conversation.