[Dashboard] Add dashboard multi-node churn test #11768
Conversation
mfitton
requested review from
rkooo567 and
edoakes
and removed request for
rkooo567
| t = threading.Thread(target=cluster_chaos_monkey, daemon=True) | ||
| t.start() |
There was a problem hiding this comment.
Could consider using a ray actor for this as well
There was a problem hiding this comment.
@edoakes I like that idea. What's the best way to force my actor to live on the head node? Otherwise I'm concerned that it'll get destroyed by a node getting taken down (although I guess it would then be restarted on a node that's still up).
There was a problem hiding this comment.
@mfitton you can create a custom resource on the head node and target that resource with Actor.options(resources=xxx).
| t = threading.Thread(target=cluster_chaos_monkey, daemon=True) | ||
| t.start() |
There was a problem hiding this comment.
@mfitton you can create a custom resource on the head node and target that resource with Actor.options(resources=xxx).
|
@mfitton lint is failing: |
edoakes
added
the
@author-action-required
…n test longer again now that I realized that was not the issue.
mfitton
added
tests-ok
Why are these changes needed?
Adds a test to the dashboard that makes sure it continues to correctly function while a thread adds and removes nodes from the cluster at random.
This test had another line to test the node summary endpoint, but that currently causes a failure. Actually fixing this issue will be the subject of another PR. For the time being, this line is omitted.
Related issue number
Checks
scripts/format.shto lint the changes in this PR.