[core] (cgroups 6/n) CgroupManager cleans up the entire cgroup hierarchy in reverse order when destroyed. #56260
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
to perform cgroup operations. Signed-off-by: irabbani <irabbani@anyscale.com>
Signed-off-by: irabbani <irabbani@anyscale.com>
instead of clone for older kernel headers < 5.7 (which is what we have in CI) Signed-off-by: irabbani <irabbani@anyscale.com>
Signed-off-by: irabbani <irabbani@anyscale.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: Ibrahim Rabbani <israbbani@gmail.com>
…irabbani/cgroups-1
Signed-off-by: irabbani <irabbani@anyscale.com>
Signed-off-by: irabbani <irabbani@anyscale.com>
Signed-off-by: irabbani <irabbani@anyscale.com>
Signed-off-by: irabbani <irabbani@anyscale.com>
Signed-off-by: irabbani <irabbani@anyscale.com>
Signed-off-by: irabbani <irabbani@anyscale.com>
Signed-off-by: irabbani <irabbani@anyscale.com>
Signed-off-by: irabbani <irabbani@anyscale.com>
Signed-off-by: irabbani <irabbani@anyscale.com>
Signed-off-by: irabbani <irabbani@anyscale.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: Ibrahim Rabbani <israbbani@gmail.com>
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: Ibrahim Rabbani <israbbani@gmail.com>
…irabbani/cgroups-1
Signed-off-by: irabbani <irabbani@anyscale.com>
fix CI. Signed-off-by: irabbani <irabbani@anyscale.com>
Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>
Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>
Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>
…et and core worker (#56285) This PR stacks on #56260. For more details about the resource isolation project see #54703. This PR removes previous code for integrating resource isolation with the raylet and core worker. It's a prerequisite for integrating the CgroupManager with the raylet. This PR is cleanup only and there is no behavior change. --------- Signed-off-by: irabbani <irabbani@anyscale.com> Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>
ZacAttack
reviewed
Sep 11, 2025
| // CgroupDriver::DeleteCgroup. | ||
| void CgroupManager::RegisterDeleteCgroup(const std::string &cgroup_path) { | ||
| cleanup_operations_.emplace_back([cgroup = cgroup_path]() { | ||
| RAY_LOG(INFO) << absl::StrFormat("Deleting all cgroup %s.", cgroup); |
Contributor
There was a problem hiding this comment.
Out of curiosity, why remove the log? (I won't insist on bringing the log back, just curious to the motivation)
Contributor
Author
There was a problem hiding this comment.
I left it in by mistake from the previous PR. I was using it to debug a test.
ZacAttack
reviewed
Sep 11, 2025
| cleanup_operations_.emplace_back([this, from_cgroup = from, to_cgroup = to]() { | ||
| Status s = this->cgroup_driver_->MoveAllProcesses(from_cgroup, to_cgroup); | ||
| if (!s.ok()) { | ||
| RAY_LOG(WARNING) << absl::StrFormat( |
Contributor
There was a problem hiding this comment.
nit: WARNING vs. ERROR?
Also, what is the potential implication of this failing? If we're unable to move processes into their appropriate cgroup, then.... what, they'll just run without the resource limits we'd otherwise like imposted? Should we keep running in that scenario or bail out?
Contributor
There was a problem hiding this comment.
I guess this line of questioning applies to delete too, but.... process move seems more concerning.
Contributor
Author
There was a problem hiding this comment.
This is during clean up. When CgroupManager starts up with a base_cgroup, it moves all processes in that cgroup into the <base_cgroup_path>/system/leaf cgroup. This is reversing by moving those processes back into the base_cgroup.
The reason for doing this is that the default cgroup path, and the most commonly anticipated pattern, will be to run this with /sys/fs/cgroup inside a container which is the root cgroup for the container.
However, the root cgroup for the container is not the actual root cgroup of the operating system. This means that it is subject to the No Internal Processes constraint. It cannot have child cgroups and a controller enabled if it has processes inside it.
The implication for clean up is that the cgroup hierarchy will not be deleted properly. This is mostly fine when it's running inside a container. Given this, maybe ERROR makes more sense?
ZacAttack
approved these changes
Sep 11, 2025
Contributor
ZacAttack
left a comment
ZacAttack
left a comment
There was a problem hiding this comment.
Synced offline, this looks good
This PR stacks on #56285. For more details about the resource isolation project see #54703. This PR integrates the CgroupManager with the raylet startup in `main.cc`. It includes - A cross-platform cgroup_manager bazel target that selectively compiles dependencies for linux/non-linux platforms. I prefer this design because it keeps targets small and compile times low and removes unnecessary ifdefs from the code. - Adds the following command-line args to the raylet binary - enable_resource_isolation - cgroup_path - system_reserved_cpu_weight - system_reserved_memory_bytes I've left comments on the PR to highlight parts that could use some feedback/discussion. --------- Signed-off-by: irabbani <irabbani@anyscale.com> Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com>
TimothySeah
added a commit
to TimothySeah/ray
that referenced
this pull request
Sep 13, 2025
…p hierarchy in reverse order when destroyed. (ray-project#56260)" This reverts commit f900567.
edoakes
added a commit
to edoakes/ray
that referenced
this pull request
Sep 14, 2025
…p hierarchy in reverse order when destroyed. (ray-project#56260)" This reverts commit f900567.
edoakes
added a commit
to edoakes/ray
that referenced
this pull request
Sep 14, 2025
…re cgroup hierarchy in reverse order when destroyed. (ray-project#56260)"" This reverts commit c4c774e.
edoakes
added a commit
to edoakes/ray
that referenced
this pull request
Sep 14, 2025
…re cgroup hierarchy in reverse order when destroyed. (ray-project#56260)"" This reverts commit c4c774e.
ZacAttack
pushed a commit
to ZacAttack/ray
that referenced
this pull request
Sep 24, 2025
…chy in reverse order when destroyed. (ray-project#56260) Signed-off-by: irabbani <irabbani@anyscale.com> Signed-off-by: Ibrahim Rabbani <israbbani@gmail.com> Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: zac <zac@anyscale.com>
marcostephan
pushed a commit
to marcostephan/ray
that referenced
this pull request
Sep 24, 2025
…chy in reverse order when destroyed. (ray-project#56260) Signed-off-by: irabbani <irabbani@anyscale.com> Signed-off-by: Ibrahim Rabbani <israbbani@gmail.com> Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: Marco Stephan <marco@magic.dev>
dstrodtman
pushed a commit
to dstrodtman/ray
that referenced
this pull request
Oct 6, 2025
…chy in reverse order when destroyed. (ray-project#56260) Signed-off-by: irabbani <irabbani@anyscale.com> Signed-off-by: Ibrahim Rabbani <israbbani@gmail.com> Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: Douglas Strodtman <douglas@anyscale.com>
justinyeh1995
pushed a commit
to justinyeh1995/ray
that referenced
this pull request
Oct 20, 2025
…chy in reverse order when destroyed. (ray-project#56260) Signed-off-by: irabbani <irabbani@anyscale.com> Signed-off-by: Ibrahim Rabbani <israbbani@gmail.com> Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
landscapepainter
pushed a commit
to landscapepainter/ray
that referenced
this pull request
Nov 17, 2025
…chy in reverse order when destroyed. (ray-project#56260) Signed-off-by: irabbani <irabbani@anyscale.com> Signed-off-by: Ibrahim Rabbani <israbbani@gmail.com> Signed-off-by: Ibrahim Rabbani <irabbani@anyscale.com> Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR stacks on #56246.
For more details about the resource isolation project see #54703.
This PR implements RAII cleanup in CgroupManager. If the CgroupManager is destroyed, it will undo all cgroup-related side-effects in reverse order. There are four kinds of side-effects
A few notes about clean up:
I've left comments in the PR outlining the most suspect aspects of the design. Happy to incorporate feedback.