Ray with Docker (ray-project#324)

* Ray with Docker

* cleanup based on comments

* rename docker user to ray-user

* add examples docker image

* working toward reliable Docker devel image

* adjust ray-user uid for Linux builds on AWS

* update documentation

* reduced dependencies for examples

* updated Docker documentation

* experimental notice on developing with Docker

README.md

@@ -49,7 +49,7 @@ estimate of pi (waiting until the computation has finished if necessary).
 
 ## Next Steps
 
-- Installation on [Ubuntu](doc/install-on-ubuntu.md), [Mac OS X](doc/install-on-macosx.md), [Windows](doc/install-on-windows.md)
+- Installation on [Ubuntu](doc/install-on-ubuntu.md), [Mac OS X](doc/install-on-macosx.md), [Windows](doc/install-on-windows.md), [Docker](doc/install-on-docker.md)
 - [Tutorial](doc/tutorial.md)
 - [About the System](doc/about-the-system.md)
 - [Using Ray on a Cluster](doc/using-ray-on-a-cluster.md)

build-docker.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+docker build -t amplab/ray:devel docker/devel
+docker build -t amplab/ray:deploy docker/deploy
+docker build -t amplab/ray:examples docker/examples

doc/install-on-docker.md

@@ -0,0 +1,106 @@
+# Installation on Docker
+
+You can install Ray on any platform that runs Docker. We do not presently publish Docker images for Ray, but you can build them yourself using the Ray distribution. Using Docker can provide a reliable way to get up and running quickly.
+
+## Install Docker
+
+The Docker Platform release is available for Mac, Windows, and Linux platforms. Please download the appropriate version from the [Docker website](https://www.docker.com/products/overview#/install_the_platform).
+
+## Clone the Ray repository
+
+```
+git clone https://github.com/amplab/ray.git
+```
+
+## Build Docker images
+
+Run the script to create Docker images.
+
+```
+cd ray
+./build-docker.sh
+```
+
+This script creates several Docker images:
+
+ * The `amplab/ray:deploy` image is a self-contained copy of code and binaries suitable for end users.
+ * The `amplab/ray:examples` adds additional libraries for running examples.
+ * Ray developers who want to edit locally on the host filesystem should use the `amplab/ray:devel` image, which allows local changes to be reflected immediately within the container. 
+
+## Launch Ray in Docker
+
+Start out by launching the deployment container.
+
+```
+docker run --shm-size=1024m -t -i amplab/ray:deploy
+```
+
+## Test if the installation succeeded
+
+To test if the installation was successful, try running some tests.
+
+```
+python test/runtest.py # This tests basic functionality.
+python test/array_test.py # This tests some array libraries.
+```
+
+You are now ready to continue with the [Tutorial](tutorial.md).
+
+## Running examples in Docker
+
+Ray includes a Docker image that includes dependencies necessary for running some of the examples. This can be an easy way to see Ray in action on a variety of workloads.
+
+Launch the examples container.
+```
+docker run --shm-size=1024m -t -i amplab/ray:examples
+```
+
+### Hyperparameter optimization
+
+
+```
+cd ~/ray/examples/hyperopt/
+python driver.py
+```
+
+See the [Hyperparameter optimization documentation](../examples/hyperopt/README.md).
+
+### Batch L-BFGS
+
+```
+cd ~/ray/examples/lbfgs/
+python driver.py
+```
+
+See the [Batch L-BFGS documentation](../examples/lbfgs/README.md).
+
+### Learning to play Pong
+
+```
+cd ~/ray/examples/rl_pong/
+python driver.py
+```
+
+See the [Learning to play Pong documentation](../examples/rl_pong/README.md).
+
+
+## Developing with Docker (Experimental)
+
+These steps apply only to Ray developers who prefer to use editing tools on the host machine while building and running Ray within Docker. If you have previously been building locally we suggest that you start with a clean checkout before building with Ray's developer Docker container.
+
+You may see errors while running `setup.sh` on Mac OS X. If you have this problem please try re-running the script. Increasing the memory of Docker's VM (say to 8GB from the default 2GB) seems to help.
+
+
+Launch the developer container.
+
+```
+docker run -v $(pwd):/home/ray-user/ray --shm-size=1024m -t -i amplab/ray:devel
+```
+
+Build Ray inside of the container.
+
+```
+cd ray
+./setup.sh
+./build.sh
+```

docker/deploy/Dockerfile

@@ -0,0 +1,17 @@
+FROM ubuntu:xenial
+RUN apt-get update
+RUN apt-get -y install apt-utils
+RUN apt-get -y install sudo
+RUN apt-get install -y git cmake build-essential autoconf curl libtool python-dev python-numpy python-pip libboost-all-dev unzip graphviz
+RUN pip install ipython typing funcsigs subprocess32 protobuf==3.0.0a2 colorama graphviz cloudpickle
+RUN adduser --gecos --ingroup ray-user --disabled-login --gecos ray-user
+RUN adduser ray-user sudo
+RUN sed -i "s|%sudo\tALL=(ALL:ALL) ALL|%sudo\tALL=NOPASSWD: ALL|" /etc/sudoers
+USER ray-user
+WORKDIR /home/ray-user
+RUN git clone https://github.com/amplab/ray
+WORKDIR /home/ray-user/ray
+RUN ./setup.sh
+RUN ./build.sh
+RUN echo '\n# Ray environment\nsource /home/ray-user/ray/setup-env.sh' >> /home/ray-user/.bashrc
+ENTRYPOINT bash

docker/devel/Dockerfile

@@ -0,0 +1,15 @@
+FROM ubuntu:xenial
+RUN apt-get update
+RUN apt-get -y install apt-utils
+RUN apt-get -y install sudo
+RUN apt-get install -y git cmake build-essential autoconf curl libtool python-dev python-numpy python-pip libboost-all-dev unzip graphviz
+RUN pip install ipython typing funcsigs subprocess32 protobuf==3.0.0a2 colorama graphviz cloudpickle
+RUN adduser --gecos --ingroup ray-user --disabled-login --gecos ray-user --uid 500
+RUN adduser ray-user sudo
+RUN sed -i "s|%sudo\tALL=(ALL:ALL) ALL|%sudo\tALL=NOPASSWD: ALL|" /etc/sudoers
+USER ray-user
+WORKDIR /home/ray-user
+RUN mkdir /home/ray-user/ray-build
+RUN mkdir /home/ray-user/ray
+RUN ln -s /home/ray-user/ray-build /home/ray-user/ray/build
+ENTRYPOINT bash

docker/examples/Dockerfile

@@ -0,0 +1,11 @@
+FROM amplab/ray:deploy
+
+# Tensorflow
+RUN pip install --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.9.0-cp27-none-linux_x86_64.whl
+
+# SciPy
+RUN pip install scipy
+
+# Gym
+RUN sudo apt-get -y install zlib1g-dev libjpeg-dev xvfb libav-tools xorg-dev python-opengl libsdl2-dev swig wget
+RUN pip install gym[atari]