[air] Add xgboost release test for silver tier(10-node case).

[xwjiang2010]

Why are these changes needed?

Add release test for air xgboost benchmark.
Will take a look at the number first running in release test environment and add a proper triggering threshold (time * 1.1).

Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[xwjiang2010]

https://buildkite.com/ray-project/release-tests-pr/builds/8985#0181f2c6-66a3-4242-a707-8d33eca2e37f

[xwjiang2010]

Another run: https://buildkite.com/ray-project/release-tests-pr/builds/8990#0181f2ef-b0be-4884-aca2-98f0cd3ec00c

[Yard1]

LGTM thanks!

[richardliaw]

nice! How do we get this onto the dashboard?

[xwjiang2010]

@Yard1 Do you know why the result shows

training_time = 1850.115315906 --> this seems too slow to me
prediction_time = 86.38269096899967 --> this seems too fast to me

There is a ray.shutdown in the middle.

[xwjiang2010]

nice! How do we get this onto the dashboard?

waiting for Kai to set something up in his PR and will just follow the practice.

[Yard1]

Actually to be 100% sure can we run the training part and prediction part as separate subprocesses?

[Yard1]

also @xwjiang2010 we are missing the EBS config, I think that may be a cause for slowdown

[Yard1]

@xwjiang2010 we probably can bring down the disk size to like 200 GBs to save a little money!

[xwjiang2010]

@Yard1 haha sounds good :)
do you know how can I know how much is being spilled?

[Yard1]

I think you can use this? https://docs.ray.io/en/latest/ray-core/objects/object-spilling.html#stats

[xwjiang2010]

training_time = 1663.45228263
prediction_time = 77.18809864599962

@Yard1

[xwjiang2010]

pretty big deviation here
run1:

2022-07-13 11:21:17,897      INFO main.py:1519 -- [RayXGBoost] Finished XGBoost training on training data with total N=260,000,000 in 822.28 seconds (701.52 pure XGBoost training time).
2022-07-13 11:21:19,560 INFO tune.py:738 -- Total run time: 827.66 seconds (826.38 seconds for the tuning loop).

run2:

[RayXGBoost] Finished XGBoost training on training data with total N=260,000,000 in 950.20 seconds (718.92 pure XGBoost training time).
2022-07-13 22:20:12,407 INFO tune.py:738 -- Total run time: 955.67 seconds (954.53 seconds for the tuning loop).

The data loading part is the cause of slowdown in 2nd run. @Yard1 , if you are OK, I am going to relax the criteria a bit more, say 1000s.

[Yard1]

That should be fine, thanks