Readiness probe must not be the same as the liveness one

[mykola-mokhnach-parloa]

Currently both liveness and readiness probes in the chart are set to check the same endpoint, which is /.

This might not work as expected in case, for example, if we need to gracefully shutdown the service.
For the time range the pod is being gracefully shut down (e.g. it has already received SIGTERM, but terminationGracePeriodSeconds has not expired yet and the service itself is still running/cleaning up) I would expect the readiness probe to return an error, so no traffic is forwarded to the given pod, while the liveness probe must still return 200, thus preventing the pod from being killed and restarted.

[mykola-mokhnach-parloa]

Linking https://www.reddit.com/r/kubernetes/comments/wayj42/what_should_readiness_liveness_probe_actually/?rdt=53287

[hocmemini]

Hi @mykola-mokhnach-parloa, opened a PR (#161) that addresses the configuration side of this issue. Wanted to flag honestly that the PR implements operator-configurable probes rather than a server-side fix to the underlying SIGTERM behavior. Operators can now point readinessProbe at a different endpoint, but livekit-server itself doesn't yet expose one that flips on SIGTERM. The full fix you originally described would also require upstream server work, which I'd want to scope as a separate issue in livekit/livekit. Happy to file that follow-up if the maintainers think it's the right move.

[eilon-decart]

Production datapoint for why this matters: with the chart's shared GET / probe, a ~7-minute outage of our (single) Redis instance failed the health endpoint on all 6 livekit-server pods simultaneously (node stats only refresh via a Redis pub/sub round-trip, so / goes 406 fleet-wide within seconds), and the liveness probe SIGTERM'd the entire SFU fleet into livekit's unbounded graceful drain — ~53 minutes of 100% signaling downtime from a Redis blip, recovered only by manually deleting the pods.

Full write-up with timeline and code references: livekit/livekit#4663.

Splitting liveness from readiness (as #161 proposes) would have turned this from a fleet-wide kill into a temporary readiness removal. Would love to see that PR move.

[feldgendler]

Opened #166 for this. Both probes become structured values that can be overridden independently (or set to ~ to omit), and the liveness default moves from GET / to a TCP check on the same port.

The default change matters as much as the configurability here: GET / returns 406 once the node's stats are more than 4s stale, and on a redis deployment those stats are refreshed by a keepalive that round-trips through redis — so a redis outage longer than about 35s fails liveness on every node simultaneously and restarts the whole fleet. livekit/livekit#4663 is a production incident where that turned a 7-minute redis blip into 53 minutes of downtime. Readiness stays on GET /, which is where staleness belongs.