[core] Nightly release test with cross AZ fault injection #57579
Conversation
Signed-off-by: joshlee <joshlee@anyscale.com>
Signed-off-by: joshlee <joshlee@anyscale.com>
Signed-off-by: joshlee <joshlee@anyscale.com>
|
Not sure what the difference is between "aws" and "aws_perf" in the env section in the yaml. |
ray-gardener
bot
added
core
There was a problem hiding this comment.
Pull Request Overview
This PR adds nightly release tests to model data workflow for cross-availability zone RPC fault tolerance testing. The tests evaluate Ray's behavior with and without RPC failure injection to ensure robustness in distributed environments.
Key changes:
- Added two new nightly tests with GCE and AWS variants for cross-AZ RPC testing
- Enhanced map_benchmark.py to support configurable concurrency parameters
- Created cluster configuration files for both GCE and AWS environments
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| release/release_data_tests.yaml | Adds two nightly tests for cross-AZ RPC fault tolerance with and without failure injection |
| release/nightly_tests/dataset/map_benchmark.py | Adds configurable concurrency parameter and refactors map_batches logic |
| release/nightly_tests/dataset/cross_az_250_350_compute_gce.yaml | GCE cluster configuration for cross-AZ testing with 250-350 workers |
| release/nightly_tests/dataset/cross_az_250_350_compute_aws.yaml | AWS cluster configuration for cross-AZ testing with 250-350 workers |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| "output of the first run as input." | ||
| ), | ||
| ) | ||
| parser.add_argument( |
There was a problem hiding this comment.
larger concurrency is needed for this workload
Sparks0219
changed the title
|
cc @alexeykudinkin and @bveeramani for a review from the data side. |
Signed-off-by: joshlee <joshlee@anyscale.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Signed-off-by: joshlee <joshlee@anyscale.com>
There was a problem hiding this comment.
Nice @Sparks0219
Next let's build some ray core cross-az tests as well :)
| min_workers: 250 | ||
| max_workers: 350 | ||
| flags: | ||
| enable_multi_az_serve: true |
There was a problem hiding this comment.
this one is not relevant, it's for ray serve workloads
There was a problem hiding this comment.
this is actually the hack we use to spread cross az, just setting allow won't actually spread, it'll just "allow"
release/nightly_tests/dataset/cross_az_250_350_compute_gce.yaml
Outdated
Show resolved
Hide resolved
Signed-off-by: joshlee <joshlee@anyscale.com>
Signed-off-by: joshlee <joshlee@anyscale.com>
5 participants
Why are these changes needed?
Created a nightly release test that tests data workflow with and without rpc fault injection. Have a gce and aws variant of the test.
Related issue number
Checks
git commit -s) in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.