[Core] Extend chaos testing utility to more closely replicate spot instance preemptions

[bveeramani]

Description

We're running a large-scale batch inference job on spot instances and trying to use as many GPUs as possible. We've observed this pattern for preemptions:

To more closely replicate this pattern, it'd be helpful if the chaos testing utility could:

1. Kill many nodes at once (e.g., ~200 above)
2. Prevent the cluster from scaling back up (to mimic spot instances being unavailable)

Use case

See above.

[bveeramani]

cc @anyscalesam

[hongchaodeng]

From slack:

we figured out a hack to mimic preemptions (decreasing the number of nodes in the compute config), so this issue isn’t urgent anymore.
We’ll still want this later so that we can test against preemptions in our release tests.