Add multinode tests by simulating multiple nodes using Docker.

[jssmith]

For testing we want to be able to simulate a Ray cluster using a number of Docker instances running on a single host. This change adds scripts to boot a cluster matching a specific configuration and to run a test script on that cluster.

Other requirements:

• Test scripts should run from the command line as well as from within the cluster environment
• One continuous integration host should simultaneously run tests from different Ray revisions

[AmplabJenkins]

Merged build finished. Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/320/
Test FAILed.

[AmplabJenkins]

Merged build finished. Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/333/
Test FAILed.

[AmplabJenkins]

Merged build finished. Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/334/
Test PASSed.

[AmplabJenkins]

Merged build finished. Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/335/
Test PASSed.

[AmplabJenkins]

Merged build finished. Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/336/
Test FAILed.

[AmplabJenkins]

Merged build finished. Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/337/
Test PASSed.

[AmplabJenkins]

Merged build finished. Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/338/
Test FAILed.

[robertnishihara]

After doing build-docker.sh, the various tests can be run from the command line with something like

python test/jenkins_tests/multi_node_docker_test.py \
    --num-nodes=5 \
    --test-script=/ray/test/jenkins_tests/multi_node_tests/test_0.py \
    --development-mode

The --development-mode flag copies the local test files into the head node's docker container so that you can edit the tests and run them without rebuilding the docker image.

[robertnishihara]

After this PR is merged, all PR's should be required to pass the Travis tests as well as the Jenkins tests (the multi-node docker stuff in this PR is only being run on Jenkins).

[robertnishihara]

The script ./test/jenkins_tests/run_multi_node_tests.sh is what is running on jenkins.

[AmplabJenkins]

Merged build finished. Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/339/
Test PASSed.

[jssmith]

Any ideas on a better way to know that Ray has started?

[robertnishihara]

Good point, now that I think about it, it should be possible to just go ahead and run the test script even if Ray hasn't started (if Ray hasn't started yet, then the call to ray.init should just retry as needed).

[robertnishihara]

I just tried it out, it works like I said with the caveat that ray.init only retries for a few seconds and then raises an exception.

[AmplabJenkins]

Merged build finished. Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/340/
Test PASSed.

[AmplabJenkins]

Merged build finished. Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/342/
Test PASSed.