[Core] (Resource Isolation 8/n) Add time based worker killing policy by Kunchd · Pull Request #61323 · ray-project/ray

Kunchd · 2026-02-25T21:14:19Z

Description

Note: This PR is a no-op. It does not switch out the existing killing policy with the new one. The new policy will be switched on when all components of the new memory monitoring system are in place.

Previously, we selected which workers to kill when under memory pressure based on the following priorities:

Workers from groups that are retriable are selected first.
Workers from the group with the largest number of workers are prioritized next.
Workers from the newest group is used to tie break.
Where group is defined by workers with the same parent task.

However, this policy suffers from the following issue:

In most cases, workers will share the same parent task in a ray cluster, so the policy becomes selecting retriable workers first and newest workers next.
The existing policy only selects a single worker to kill each time memory monitor is triggered. This causes the system to be unable to terminate workers fast enough to keep the system from being impacted by the kernel OOM killer when under significant memory pressure.

To address these issues, we introduce the new simplified and effective time based worker killing policy.

The new policy simply selects which workers to kill based on:

workers that are retriable are prioritized first
newer workers are used to tie-break (since we want to preserve work of longer running workers).
Finally, the new policy will continue killing workers until we've been put back under the ray memory limit threshold, ensuring we are killing aggressively enough to relieve the system of memory pressure.

Related issues

Additional information

Signed-off-by: davik <davik@anyscale.com>

gemini-code-assist

Code Review

This pull request introduces a new time-based worker killing policy for resource isolation. The new policy prioritizes killing retriable workers, and then newer workers, until memory usage is below a certain threshold. The changes include the new policy implementation, corresponding tests, and build file updates. The code is well-structured and the tests are comprehensive. I have one suggestion to improve the readability of the sorting logic in the policy implementation.

Signed-off-by: davik <davik@anyscale.com>

israbbani

Looks good. A few questions and nit.

israbbani · 2026-03-03T19:34:35Z

        "//src/ray/common:lease",
        "//src/ray/protobuf:common_cc_proto",
        "//src/ray/raylet:worker_interface",
+        "//src/ray/util:compat",


still annoyed that we have to include compat for pid_t

Signed-off-by: davik <davik@anyscale.com>

cursor

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Kunchd · 2026-03-09T01:32:04Z

@edoakes Could you help me merge this. Thanks!

…61323) ## Description **Note**: This PR is a no-op. It does not switch out the existing killing policy with the new one. The new policy will be switched on when all components of the new memory monitoring system are in place. Previously, we selected which workers to kill when under memory pressure based on the following priorities: 1. Workers from groups that are retriable are selected first. 2. Workers from the group with the largest number of workers are prioritized next. 3. Workers from the newest group is used to tie break. Where group is defined by workers with the same parent task. However, this policy suffers from the following issue: * In most cases, workers will share the same parent task in a ray cluster, so the policy becomes selecting retriable workers first and newest workers next. * The existing policy only selects a single worker to kill each time memory monitor is triggered. This causes the system to be unable to terminate workers fast enough to keep the system from being impacted by the kernel OOM killer when under significant memory pressure. To address these issues, we introduce the new simplified and effective time based worker killing policy. The new policy simply selects which workers to kill based on: 1. workers that are retriable are prioritized first 2. newer workers are used to tie-break (since we want to preserve work of longer running workers). Finally, the new policy will continue killing workers until we've been put back under the ray memory limit threshold, ensuring we are killing aggressively enough to relieve the system of memory pressure. ## Related issues ## Additional information --------- Signed-off-by: davik <davik@anyscale.com> Co-authored-by: davik <davik@anyscale.com> Co-authored-by: Ibrahim Rabbani <irabbani@anyscale.com>

…me killing policy (#62643) ## Description This PR replaces the existing by group killing policy with the time based worker killing policy. **Note:** This will introduces a behavioral change in the next version, where: 1. We no longer prioritize killing large worker groups first. 2. We can now select multiple workers to kill at a time when memory pressure is detected. The existing by group killing policy selects the worker to kill based on the group size (where a group is defined by the number workers belonging to the same owner), retry-ability, and submission time. Additionally, it also only selects a single worker to killing at a time. This is problematic in situations where eliminating a single worker may be insufficient to bring the system back below the memory threshold, failing to reduce memory pressure. Sorting by the group size also doesn't achieve our goal of always preserving the tasks that has completed more work (estimated by elapsed time since start of task execution). Finally, in most workloads there are typically only a single owner for tasks, thus sorting by groups is redundant. The new killing policy addresses these issues by always prioritizing workers with longer execution duration in order to preserve more work, and it can kill multiple workers when necessary to put us back under the killing threshold. ## Additional information PR that introduced the time based killing policy: #61323 --------- Signed-off-by: davik <davik@anyscale.com> Co-authored-by: davik <davik@anyscale.com>

…ry to create new memory monitoring system (#62705) ## Description This PR creates the wiring needed to support a memory monitoring system with multiple memory monitors and sets up the cgroup constraints needed for the resource isolation design described below. Specifically this PR creates the `multi_monitor_factory` responsible for creating the correct combination of memory monitors depending on user configuration. Additionally it modifies the `cgroup_manager` to set up the `memory.high` constraint needed for providing resource isolation. For more detail on the expected configuration, please see the descriptions of resource isolation below. **Note**: This PR only introduces the wiring needed for the system below for ease of review. ### Ray's Memory Model Before we discuss the problem resource isolation is attempting to resolve, let's start with an overview of ray's memory model. At a high level, ray's memory usage on each node can be broken down into three parts. <img width="872" height="284" alt="image" src="https://github.com/user-attachments/assets/1e786a02-d2e7-442e-acf0-83f6ace18359" /> * System memory: the memory usage of ray system processes. This includes the raylet used to manage all ray processes running on the node and the agents responsible for emitting observability metrics (and more...). * Object store memory: the shared memory used for storing the objects produced by the user function. This includes the objects put into the object store via `ray.put`, and the objects you return from a ray function. * User memory: the heap memory used by all the workers running user defined tasks (including actor tasks) on the host. The following sections will focus on isolating the user memory segment from impacting the system processes while enhancing user slice performance under even memory oversubscription. ### The Problem Ray currently lacks a means to isolate user application processes from system processes that are critical to cluster health. This results in the following problems: * Under significant resource contention caused by workload oversubscription, critical processes such as the raylet can become starved for resources, which snow balls into raylet stalling and ultimately leading to node deaths. * When the host itself is under memory contention, the kernel OOM killer will trigger, killing arbitrary processes. As the kernel OOM killer is not workload aware, this may result in significant work lost. ### Why is the existing solution insufficient Our goal is to provide two guarantees when user run workloads on Ray. * The system can continue to make process regardless of the resource usage of the user tasks. * User workloads should continue to make progress even when under resource contention and OOMs. To address the first issue, the existing system introduces the `ThresholdMemoryMonitor`. This monitor works by periodically polling the host system's memory usage information to determine the current state of memory utilization, and it will kick off Ray's oom killing policy when the utilization exceeds a certain threshold. The hopes of this system is that we always reserve some amount of free memory (`total_memory - threshold`) for the system processes on the host to make progress and will trigger the Ray OOM killer to kill off workers if the threshold is exceeded. To address the second issue, the existing Ray OOM kill policy will select a single worker to kill each time the threshold is exceeded. This selection is based on the time of the start of execution and attempt to preserve worker that runs longer. However, we have observed that the poll based memory monitor alone is insufficient for enforcing the memory threshold. This is due to the following issue: * The poll based model can potentially miss memory burst events between intervals. * The killing policy may fail to kill aggressively enough to put us back under the threshold. Overall, the existing solution fails to guarantee that the workload memory usage won't impact the system processes. ### Solution/What we introduce Cgroups to the rescue! Unlike our existing memory monitoring system which needs to constantly poll the host system in hopes that we don't miss a memory hungry process, cgroups provides us with tools that enforces memory usage limits for groups of processes. <img width="1084" height="848" alt="image" src="https://github.com/user-attachments/assets/1a83ffae-b1c1-4ef6-8ccc-17000d0c80b6" /> With this tool, let's first tackle the problem of protecting critical system processes from memory hungry workers. Let's return to our previously described memory model. The system memory slice will remain relatively consistent as Ray is responsible for the system processes, so setting aside a fixed amount of memory for it should be sufficient. The object store and user application memory usage are both dynamic and dependent on user workloads, so it is natural to put both under an upper bound memory constraint that prevents them from eating into system reserved memory. This all seems great, perhaps a little too good to be true. And unfortunately, cgroup's memory model decided to throw us a [curve ball](https://docs.kernel.org/admin-guide/cgroup-v2.html#memory-ownership). Since object store is memory shared between the raylet in the system slice and user applications, it can belong to either the system or user slice. So, we address this issue with two separate memory monitors. * In the top diagram, we consider the case where both object store memory and user application remain within the user cgroup. In this scenario, we set a `memory.high` upper bound constraint that prevents the two from exceeding the memory limit, and kick in the event memory monitor to select workers to kill when it is met. * In the bottom diagram, we consider the case where a portion of the object store memory may have escaped the user cgroup. Since this usage is no longer visible to the user cgroup enforcing the `memory.high`, we introduce the threshold memory monitor to catch this by monitoring the system wide object store usage and the user application usage. This way, we can still catch the scenario where object store memory has escaped our user cgroup protection. So, the issue of protecting the system processes is resolved. What about ensuring workloads can continue to make progress even under resource contention. This is accomplished with our design as well as it ensures that the ray OOM killer will trigger before the kernel OOM killer. This is particularly useful as the ray OOM killer is workload aware and selects workers to kill based on time since start of execution to approximate killing the worker with the least amount of work done. ### What will change At the completion of this project, when resource isolation is enabled, the above discussed memory monitoring system will be enabled. When resource isolation is disabled, we will maintain the same behavior as before. However, the new killing policy will be applied to both the existing memory monitoring system with resource isolation disabled, and our new memory monitoring system. ### Performance So how well do the changes actually protect Ray from kernel OOMs which are detrimental for performance? Here we show our experiments across simulated (first 4) and real world (last 3) memory heavy workloads. <img width="1053" height="606" alt="image" src="https://github.com/user-attachments/assets/d89971fe-5a84-4f03-abef-3413ad1e0c1b" /> Additionally, we have also observed while running the workloads above that memory throttling mode successfully eliminates node failures (caused by memory starvation of critical ray system processes) compared to the existing monitoring system without resource isolation, where we typically observe node failures through out the video object detection workload. ## Additional information * PR which introduced the new worker killing policy: #61323 * PR which introduced the pressure memory monitor: #61361 * PR which introduced the event memory monitor on memory throttling mode: #62060 --------- Signed-off-by: davik <davik@anyscale.com> Co-authored-by: davik <davik@anyscale.com>

…me killing policy (ray-project#62643) ## Description This PR replaces the existing by group killing policy with the time based worker killing policy. **Note:** This will introduces a behavioral change in the next version, where: 1. We no longer prioritize killing large worker groups first. 2. We can now select multiple workers to kill at a time when memory pressure is detected. The existing by group killing policy selects the worker to kill based on the group size (where a group is defined by the number workers belonging to the same owner), retry-ability, and submission time. Additionally, it also only selects a single worker to killing at a time. This is problematic in situations where eliminating a single worker may be insufficient to bring the system back below the memory threshold, failing to reduce memory pressure. Sorting by the group size also doesn't achieve our goal of always preserving the tasks that has completed more work (estimated by elapsed time since start of task execution). Finally, in most workloads there are typically only a single owner for tasks, thus sorting by groups is redundant. The new killing policy addresses these issues by always prioritizing workers with longer execution duration in order to preserve more work, and it can kill multiple workers when necessary to put us back under the killing threshold. ## Additional information PR that introduced the time based killing policy: ray-project#61323 --------- Signed-off-by: davik <davik@anyscale.com> Co-authored-by: davik <davik@anyscale.com>

…ry to create new memory monitoring system (ray-project#62705) ## Description This PR creates the wiring needed to support a memory monitoring system with multiple memory monitors and sets up the cgroup constraints needed for the resource isolation design described below. Specifically this PR creates the `multi_monitor_factory` responsible for creating the correct combination of memory monitors depending on user configuration. Additionally it modifies the `cgroup_manager` to set up the `memory.high` constraint needed for providing resource isolation. For more detail on the expected configuration, please see the descriptions of resource isolation below. **Note**: This PR only introduces the wiring needed for the system below for ease of review. ### Ray's Memory Model Before we discuss the problem resource isolation is attempting to resolve, let's start with an overview of ray's memory model. At a high level, ray's memory usage on each node can be broken down into three parts. <img width="872" height="284" alt="image" src="https://github.com/user-attachments/assets/1e786a02-d2e7-442e-acf0-83f6ace18359" /> * System memory: the memory usage of ray system processes. This includes the raylet used to manage all ray processes running on the node and the agents responsible for emitting observability metrics (and more...). * Object store memory: the shared memory used for storing the objects produced by the user function. This includes the objects put into the object store via `ray.put`, and the objects you return from a ray function. * User memory: the heap memory used by all the workers running user defined tasks (including actor tasks) on the host. The following sections will focus on isolating the user memory segment from impacting the system processes while enhancing user slice performance under even memory oversubscription. ### The Problem Ray currently lacks a means to isolate user application processes from system processes that are critical to cluster health. This results in the following problems: * Under significant resource contention caused by workload oversubscription, critical processes such as the raylet can become starved for resources, which snow balls into raylet stalling and ultimately leading to node deaths. * When the host itself is under memory contention, the kernel OOM killer will trigger, killing arbitrary processes. As the kernel OOM killer is not workload aware, this may result in significant work lost. ### Why is the existing solution insufficient Our goal is to provide two guarantees when user run workloads on Ray. * The system can continue to make process regardless of the resource usage of the user tasks. * User workloads should continue to make progress even when under resource contention and OOMs. To address the first issue, the existing system introduces the `ThresholdMemoryMonitor`. This monitor works by periodically polling the host system's memory usage information to determine the current state of memory utilization, and it will kick off Ray's oom killing policy when the utilization exceeds a certain threshold. The hopes of this system is that we always reserve some amount of free memory (`total_memory - threshold`) for the system processes on the host to make progress and will trigger the Ray OOM killer to kill off workers if the threshold is exceeded. To address the second issue, the existing Ray OOM kill policy will select a single worker to kill each time the threshold is exceeded. This selection is based on the time of the start of execution and attempt to preserve worker that runs longer. However, we have observed that the poll based memory monitor alone is insufficient for enforcing the memory threshold. This is due to the following issue: * The poll based model can potentially miss memory burst events between intervals. * The killing policy may fail to kill aggressively enough to put us back under the threshold. Overall, the existing solution fails to guarantee that the workload memory usage won't impact the system processes. ### Solution/What we introduce Cgroups to the rescue! Unlike our existing memory monitoring system which needs to constantly poll the host system in hopes that we don't miss a memory hungry process, cgroups provides us with tools that enforces memory usage limits for groups of processes. <img width="1084" height="848" alt="image" src="https://github.com/user-attachments/assets/1a83ffae-b1c1-4ef6-8ccc-17000d0c80b6" /> With this tool, let's first tackle the problem of protecting critical system processes from memory hungry workers. Let's return to our previously described memory model. The system memory slice will remain relatively consistent as Ray is responsible for the system processes, so setting aside a fixed amount of memory for it should be sufficient. The object store and user application memory usage are both dynamic and dependent on user workloads, so it is natural to put both under an upper bound memory constraint that prevents them from eating into system reserved memory. This all seems great, perhaps a little too good to be true. And unfortunately, cgroup's memory model decided to throw us a [curve ball](https://docs.kernel.org/admin-guide/cgroup-v2.html#memory-ownership). Since object store is memory shared between the raylet in the system slice and user applications, it can belong to either the system or user slice. So, we address this issue with two separate memory monitors. * In the top diagram, we consider the case where both object store memory and user application remain within the user cgroup. In this scenario, we set a `memory.high` upper bound constraint that prevents the two from exceeding the memory limit, and kick in the event memory monitor to select workers to kill when it is met. * In the bottom diagram, we consider the case where a portion of the object store memory may have escaped the user cgroup. Since this usage is no longer visible to the user cgroup enforcing the `memory.high`, we introduce the threshold memory monitor to catch this by monitoring the system wide object store usage and the user application usage. This way, we can still catch the scenario where object store memory has escaped our user cgroup protection. So, the issue of protecting the system processes is resolved. What about ensuring workloads can continue to make progress even under resource contention. This is accomplished with our design as well as it ensures that the ray OOM killer will trigger before the kernel OOM killer. This is particularly useful as the ray OOM killer is workload aware and selects workers to kill based on time since start of execution to approximate killing the worker with the least amount of work done. ### What will change At the completion of this project, when resource isolation is enabled, the above discussed memory monitoring system will be enabled. When resource isolation is disabled, we will maintain the same behavior as before. However, the new killing policy will be applied to both the existing memory monitoring system with resource isolation disabled, and our new memory monitoring system. ### Performance So how well do the changes actually protect Ray from kernel OOMs which are detrimental for performance? Here we show our experiments across simulated (first 4) and real world (last 3) memory heavy workloads. <img width="1053" height="606" alt="image" src="https://github.com/user-attachments/assets/d89971fe-5a84-4f03-abef-3413ad1e0c1b" /> Additionally, we have also observed while running the workloads above that memory throttling mode successfully eliminates node failures (caused by memory starvation of critical ray system processes) compared to the existing monitoring system without resource isolation, where we typically observe node failures through out the video object detection workload. ## Additional information * PR which introduced the new worker killing policy: ray-project#61323 * PR which introduced the pressure memory monitor: ray-project#61361 * PR which introduced the event memory monitor on memory throttling mode: ray-project#62060 --------- Signed-off-by: davik <davik@anyscale.com> Co-authored-by: davik <davik@anyscale.com>

Add time based worker killing policy

c9c1b33

Signed-off-by: davik <davik@anyscale.com>

Kunchd requested a review from a team as a code owner February 25, 2026 21:14

gemini-code-assist Bot reviewed Feb 25, 2026

View reviewed changes

Comment thread src/ray/raylet/worker_killing_policy_by_time.cc

cursor Bot reviewed Feb 25, 2026

View reviewed changes

Comment thread src/ray/raylet/tests/worker_killing_policy_by_time_test.cc

Avoid pid hash conflict in time based kp test

63ab866

Signed-off-by: davik <davik@anyscale.com>

Kunchd added the go add ONLY when ready to merge, run all tests label Feb 25, 2026

cursor Bot reviewed Feb 25, 2026

View reviewed changes

Comment thread src/ray/raylet/tests/util.h

Comment thread src/ray/raylet/worker_killing_policy_by_time.cc Outdated

davik and others added 2 commits February 25, 2026 23:08

Address debug string off by one error

92fca06

Signed-off-by: davik <davik@anyscale.com>

Merge branch 'master' into time_based_kp

5bbb2a5

ray-gardener Bot added the community-contribution Contributed by the community label Feb 26, 2026

Kunchd added core Issues that should be addressed in Ray Core and removed community-contribution Contributed by the community labels Feb 26, 2026

israbbani approved these changes Mar 4, 2026

View reviewed changes

davik and others added 2 commits March 4, 2026 22:40

Clarify cause of inability to find process memory usage

fdc028f

Signed-off-by: davik <davik@anyscale.com>

Merge branch 'master' into time_based_kp

a6f8389

israbbani approved these changes Mar 6, 2026

View reviewed changes

Merge branch 'master' into time_based_kp

54fc6e3

israbbani enabled auto-merge (squash) March 6, 2026 16:59

github-actions Bot disabled auto-merge March 6, 2026 17:00

cursor Bot reviewed Mar 6, 2026

View reviewed changes

Comment thread src/ray/raylet/worker_killing_policy_by_time.cc

Comment thread src/ray/raylet/tests/worker_killing_policy_by_time_test.cc

richardliaw merged commit be18bcd into ray-project:master Mar 10, 2026
7 checks passed

Kunchd mentioned this pull request Apr 15, 2026

[Core] (Resource Isolation 12/n) Switch group killing policy to by time killing policy #62643

Merged

Kunchd mentioned this pull request Apr 22, 2026

[Core] (Resource Isolation 13/n) Introduce Multi Memory Monitor Factory to create new memory monitoring system #62705

Merged

Kunchd deleted the time_based_kp branch September 10, 2026 21:58

Navigation Menu

Uh oh!

[Core] (Resource Isolation 8/n) Add time based worker killing policy - #61323

[Core] (Resource Isolation 8/n) Add time based worker killing policy#61323
richardliaw merged 7 commits into
ray-project:masterfrom
Kunchd:time_based_kp

Kunchd commented Feb 25, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

israbbani left a comment

Uh oh!

israbbani Mar 3, 2026

Uh oh!

Kunchd Mar 4, 2026

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Kunchd commented Mar 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Kunchd commented Feb 25, 2026

Description

Related issues

Additional information

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

israbbani left a comment

Choose a reason for hiding this comment

Uh oh!

israbbani Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Kunchd Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Kunchd commented Mar 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants