From 0db9f0292337eda9fc786956f1cce336424ce50c Mon Sep 17 00:00:00 2001 From: Jack Decker <24392469+jackowfish@users.noreply.github.com> Date: Tue, 11 Aug 2026 11:40:13 -0400 Subject: [PATCH 1/2] docs: explain traffic drain gap after SIGTERM --- .../services/web-service.mdx | 4 ++++ .../configure/zero-downtime-deployments.mdx | 22 +++++++++++++++---- 2 files changed, 22 insertions(+), 4 deletions(-) diff --git a/applications/configuration-as-code/services/web-service.mdx b/applications/configuration-as-code/services/web-service.mdx index b37f2415..41a2c244 100644 --- a/applications/configuration-as-code/services/web-service.mdx +++ b/applications/configuration-as-code/services/web-service.mdx @@ -220,6 +220,10 @@ readinessCheck: initialDelaySeconds: 15 ``` + +A failing readiness check does not stop traffic instantly. Requests keep arriving for at least 1 to 5 seconds while routing catches up, so keep serving during that window. See [Graceful Shutdown](/applications/configure/zero-downtime-deployments#web-services). + + ### `startupCheck` `object` diff --git a/applications/configure/zero-downtime-deployments.mdx b/applications/configure/zero-downtime-deployments.mdx index ff2efcaf..7f941d3f 100644 --- a/applications/configure/zero-downtime-deployments.mdx +++ b/applications/configure/zero-downtime-deployments.mdx @@ -63,10 +63,24 @@ The termination grace period can be configured in the **Advanced** tab. Web services will continue to receive traffic until they exit, unless you configure the readiness probe to fail. The recommended graceful shutdown sequence is: -1. When `SIGTERM` is received, immediately return a `500`-level response on your health check endpoint to stop receiving new traffic. -2. Close the server to prevent additional connections. -3. Drain all existing connections before the grace period ends. -4. Exit gracefully after connections are drained. +1. When `SIGTERM` is received, immediately return a `500`-level response on your health check endpoint to signal that the instance should stop receiving new traffic. +2. Keep accepting new requests for a short drain window while routing catches up (see below), and start draining the connections you already hold. +3. Close the server to prevent additional connections once the drain window has passed. +4. Drain all existing connections before the grace period ends, then exit gracefully. + + +Do not close your server the moment `SIGTERM` arrives. Failing the health check does not stop traffic instantly, so an instance that stops accepting connections right away will refuse requests that are still on their way to it, which surfaces as `502`s. + + +#### Why traffic keeps arriving after SIGTERM + +Routing is updated in the background, not at the instant your health check starts failing. Once the router picks up that change it stops sending requests to the instance, which leaves a gap of at least 1 to 5 seconds after the first failing health check. `SIGTERM` is delivered on its own path, so it has no fixed ordering against the routing change. + +Size your drain window to cover that gap. A window of 5 to 15 seconds is a reasonable starting point, and the termination grace period must be longer than the drain window plus your slowest request. + + +For the underlying mechanics, see step 3 of [Pod termination flow](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination-flow) in the Kubernetes documentation. + ### Workers From 86bc128c00db92243136d8803941b718360d8af3 Mon Sep 17 00:00:00 2001 From: Jack Decker <24392469+jackowfish@users.noreply.github.com> Date: Tue, 11 Aug 2026 11:51:58 -0400 Subject: [PATCH 2/2] docs: clarify routing removal is independent of SIGTERM --- applications/configure/zero-downtime-deployments.mdx | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/applications/configure/zero-downtime-deployments.mdx b/applications/configure/zero-downtime-deployments.mdx index 7f941d3f..1d3244b3 100644 --- a/applications/configure/zero-downtime-deployments.mdx +++ b/applications/configure/zero-downtime-deployments.mdx @@ -61,20 +61,22 @@ The termination grace period can be configured in the **Advanced** tab. ### Web Services -Web services will continue to receive traffic until they exit, unless you configure the readiness probe to fail. The recommended graceful shutdown sequence is: +Web services keep receiving traffic until they are taken out of the routing, which does not happen the moment they start shutting down. The recommended graceful shutdown sequence is: -1. When `SIGTERM` is received, immediately return a `500`-level response on your health check endpoint to signal that the instance should stop receiving new traffic. +1. When `SIGTERM` is received, immediately return a `500`-level response on your health check endpoint to mark the instance as unavailable. 2. Keep accepting new requests for a short drain window while routing catches up (see below), and start draining the connections you already hold. 3. Close the server to prevent additional connections once the drain window has passed. 4. Drain all existing connections before the grace period ends, then exit gracefully. -Do not close your server the moment `SIGTERM` arrives. Failing the health check does not stop traffic instantly, so an instance that stops accepting connections right away will refuse requests that are still on their way to it, which surfaces as `502`s. +Do not close your server the moment `SIGTERM` arrives. Traffic does not stop instantly, so an instance that stops accepting connections right away will refuse requests that are still on their way to it, which surfaces as `502`s. #### Why traffic keeps arriving after SIGTERM -Routing is updated in the background, not at the instant your health check starts failing. Once the router picks up that change it stops sending requests to the instance, which leaves a gap of at least 1 to 5 seconds after the first failing health check. `SIGTERM` is delivered on its own path, so it has no fixed ordering against the routing change. +When an instance is replaced, the orchestrator starts two things at around the same time: it sends `SIGTERM` to your process, and it begins taking the instance out of the routing. Neither one waits on the other, and neither one waits on your health check. Your `500`-level response marks the instance as unavailable, but it is not the switch that stops traffic. + +Removal from the routing takes effect only once the router picks up the change, at least 1 to 5 seconds later. Every request dispatched before that still arrives at the instance, including requests sent after your process has already received `SIGTERM`. Size your drain window to cover that gap. A window of 5 to 15 seconds is a reasonable starting point, and the termination grace period must be longer than the drain window plus your slowest request.