[tune] Support restoring from S3 checkpoints

[richardliaw]

Consider supporting some subset of the following:

1. Trial Restore from S3 url.
2. Experiment restoring from S3 url.

cc @hartikainen feel free to add more details

[ericl]

Would this be a matter of rsyncing the state back when starting (since I think the checkpoint data is already synced up to s3?)

[richardliaw]

Hm, I think so. It probably isn't necessary to support `Trial(restore=s3_path)` right?

…

On Fri, Jan 11, 2019 at 7:47 AM Eric Liang ***@***.***> wrote: Would this be a matter of rsyncing the state back when starting (since I think the checkpoint data is already synced up to s3?) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3747 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEUc5SxvfSM0HiEwmGA7F7FhMAIgEdDXks5vCLItgaJpZM4Z67J9> .

[hartikainen]

For me, it would be extremely useful if I could:

1. Start an experiment from scratch on autoscaler

• ray exec ./config/ray-autoscaler.yaml "python -m examples.development.main --mode=gce --universe=gym --domain=Swimmer --task=Default --exp-name=${EXPERIMENT_NAME} --num-samples 3 --upload-dir=gs://${GS_BUCKET}/ray/results" --start --stop --tmux

2. Later, restore the same experiment on autoscaler

• ray exec ./config/ray-autoscaler.yaml "python -m examples.development.main --mode=gce --universe=gym --domain=Swimmer --task=Default --exp-name=${NEW_EXPERIMENT_NAME} --num-samples 3 --upload-dir=gs://${GS_BUCKET}/ray/results --restore=gs://${GS_BUCKET}/ray/results/${EXPERIMENT_NAME}" --start --stop --tmux

[richardliaw]

Should be closed by #4450.