Switch EC2 example config to use AWS deep learning AMI + latest Ray wheel

[ericl]

What do these changes do?

This is the fix-up PR for #1311

[ericl]

I'm not sure why the subnets list is empty until you do this; I ran into this problem when trying to use the script in a region I hadn't used before (us-west-2).

[AmplabJenkins]

Merged build finished. Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2821/
Test PASSed.

[robertnishihara]

Would it work if we get rid of sudo and add in --user

[robertnishihara]

Thanks @ericl this looks good.

Separately, what is the best way for developers to use this? E.g., should developers make a new AMI with Ray installed via python setup.py develop and then change the config to use that AMI?

Also, for development, would it make sense to pre-populate a workers.txt file on the head node? And the start_worker.sh/stop_worker.sh/upgrade.sh scripts?

[robertnishihara]

I left one comment. Feel free to address it or merge anyway.

[ericl]

@robertnishihara made that change, it seems to work.

Separately, what is the best way for developers to use this? E.g., should developers make a new AMI with Ray installed via python setup.py develop and then change the config to use that AMI?

Yeah, I snapshotted a new AMI that had my repo pre-cloned.

Also, for development, would it make sense to pre-populate a workers.txt file on the head node? And the start_worker.sh/stop_worker.sh/upgrade.sh scripts?

What I've been doing is adding a git checkout <my_sha> in the init commands and then running ray create_or_update, which updates the nodes. When I need to force restart the nodes I add some dummy init command (e.g. "echo foo") that triggers a ray restart on all nodes.

[AmplabJenkins]

Merged build finished. Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2823/
Test PASSed.

[robertnishihara]

It's got to be unrelated to this PR, but this is the first time I saw the error

======================================================================
FAIL: testCleanupOnDriverExitManyRedisShards (__main__.MonitorTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/monitor_test.py", line 88, in testCleanupOnDriverExitManyRedisShards
    self._testCleanupOnDriverExit(num_redis_shards=5)
  File "test/monitor_test.py", line 79, in _testCleanupOnDriverExit
    self.assertEqual((0, 1), StateSummary()[:2])
AssertionError: Tuples differ: (0, 1) != (4, 3)

First differing element 0:
0
4

- (0, 1)
+ (4, 3)

cc @concretevitamin

[robertnishihara]

Filed an issue at #1332.