[autoscaler] AWS Autoscaler Improvements

[pdames]

This issue seeks to consolidate and track an initial set of proposed improvements to the AWS Autoscaler. All design docs and pull requests that support these proposals will be linked here as they become available. Also, any relevant conversations or concerns related to these proposals should be tracked here as well.

These proposals have naturally arisen in the course of using Ray internally at Amazon, and focus primarily on foundational improvements to AWS Autoscaler security, testability, usability, and integration with existing AWS cloud configuration capabilities.

In particular, to make fully automated AWS Ray cluster deployments predictable and reliable at scale, we would like to ensure that the AWS Autoscaler is idempotent and deterministic during cluster config application, and guard this behavior against regression.

To achieve these goals, we would like to work with the open source community to deliver the following priority-ordered improvements, where each improvement is typically expected to be fulfilled by a single pull request:

1. Honor Separate Head and Worker Node Subnet IDs: Fix an existing issue that allows users to specify different head/worker subnets but then ignores the worker subnet. This also introduces foundational changes to allow separate EC2 security groups to exist for head and worker nodes, together with a basic stubbed unit test suite for the AWS Autoscaler. Subsequent improvements will build on top of this foundation.

2. AWS Config Bootstrap Plugin Support: To unblock a broad range of one-off user requirements, allow users to specify custom callbacks to be invoked during config bootstrapping. Minimally, users should be able to provide their own pre-bootstrap and post-bootstrap python scripts to be invoked before/after config bootstrapping. Maximally, users should be able to override either a subset or all of the default config bootstrapping workflow.

3. EC2 Security Group Inbound Rule Whitelists: Support common enterprise network security compliance use cases to ensure that inbound AWS EC2 instance connections are restricted to a known set of internal subnets, security groups, and/or prefix list IDs. These changes will also support idempotent, deterministic whitelist re-application to enable automatic correction of cluster configuration drift. We will consider the trade-offs related to adding this feature as either an example config bootstrap plugin, or as part of the default config bootstrapping workflow.

4. Disable Autoscaler Config Cache: Ensures that cluster config is applied deterministically, regardless of the state of the host applying configuration. We would like to minimally provide the option to disable the local config cache, and maximally remove the local config cache altogether.

5. EC2 Security Group Tagging: To better facilitate both automated and manual cluster security group discovery, security groups will have tags applied to track the cluster(s) that they are currently servicing. Minimally, these tags will have cluster drift detection and idempotent correction applied during config bootstrapping. Maximally, these tags will have ongoing, eventually consistent cluster drift detection and idempotent correction applied.

6. CloudFormation Template Support: To support a broader range of expected AWS account config preconditions (e.g. create an SQS queue for a Ray cluster to poll, grant the cluster permissions to read/delete messages from it, etc.), we would like to add the ability to deploy user-specified CloudFormation stacks prior to bootstrapping a Ray cluster. We will consider the trade-offs related to adding this feature as either an example config bootstrap plugin, or as part of the default config bootstrapping workflow.

7. Add More Strongly Typed Head and Worker Node Config: To resolve ambiguity about which boto3 models Ray can successfully honor during cluster creation, we would like to introduce stronger typing to quickly catch unsupported "head_node" and "worker_nodes" configuration options instead of ignoring them or launching misconfigured clusters.

8. EC2 Launch Template Support: To further simplify head and worker node configuration options, and to reduce the need to write common cluster post-configuration steps as setup commands, we would like to investigate tighter integration of EC2 launch templates with head and worker node config.

9. EC2 Security Group Outbound Rule Whitelists: These changes build on top of the inbound rule whitelists introduced in [2], and serve similar enterprise network security compliance use cases. We will consider the trade-offs related to adding this feature as either an example config bootstrap plugin, or as part of the default config bootstrapping workflow.

10. AWS Autoscaler Test Suite Coverage and Maintainability Improvements: Extend the AWS Autoscaler test suite to ensure that we have sufficient code coverage across all pre-existing AWS Autoscaler features not touched by the above changes. We will also ensure that we have created sufficient tests to both vet, and protect, idempotent and deterministic autoscaling cluster config application. We will refactor this test suite as required to improve long-term maintainability, and any additional issues discovered while improving test coverage will be appended to this issue for tracking.

[ericl]

These sound like useful improvements, thanks! For a couple items like "CloudFormation Template" and various "Security Whitelists", I am wondering if this is something we can support in a more pluggable way? For example, have a validation or setup callback that the user can configure and point to arbitrary python functions.

The reason for making these pluggable instead of a built in feature would be (1) support more possible use cases, and (2) avoid needing to maintain autoscaler features only useful to a small subset of users.

[pdames]

These sound like useful improvements, thanks! For a couple items like "CloudFormation Template" and various "Security Whitelists", I am wondering if this is something we can support in a more pluggable way? For example, have a validation or setup callback that the user can configure and point to arbitrary python functions.

The reason for making these pluggable instead of a built in feature would be (1) support more possible use cases, and (2) avoid needing to maintain autoscaler features only useful to a small subset of users.

Thanks for the feedback! Reason (1) seems like reason enough to include this feature in our milestones above. Reason (2) seems more speculative, but could very well be correct both today and long-term. Do we have mechanisms in place to gauge user engagement with features, or to figure out which features to deliver next? In this case, it would be good to know how many users have run either one-time, or ongoing, manual AWS account configuration or CloudFormation stack deployments prior to deploying their Ray clusters. It would also be good to know how many users deploy manually configured security groups or subnets with their Ray clusters. This could better inform our decision to either include first-class support for these features within AWS Autoscaler config, or to simply illustrate how to do it via a custom plugin.

Either way, we definitely want to try to keep the ongoing AWS Autoscaler feature maintenance burden as low as possible going forward, and would ideally like to close out this issue in a way that both produces a more feature-complete and lower-maintenance AWS Autoscaler code path.

[pdames]

I've revised the list above to reflect both this request and a separate improvement for EC2 security group tagging discussed in #8374.

[pdames]

Additional improvements related specifically to CloudWatch integration are available at #8967.

[stale]

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

• If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
• If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.

[stale]

Hi again! The issue will be closed because there has been no more activity in the 14 days since the last message.

Please feel free to reopen or open a new issue if you'd still like it to be addressed.

Again, you can always ask for help on our discussion forum or Ray's public slack channel.

Thanks again for opening the issue!

[pdames]

Can we re-open this issue?