Docker Updates (ray-project#308)

* new path for python build * add flag * build tar using git archive * no exit from start_ray.sh * update Docker instructions * update build docker script * add git revision * fix typo * bug fixes and clarifications * mend * add objectmanager ports to docker instructions * rewording * Small updates to documentation.
denis-xiao · Mar 1, 2017 · ad4b03b · ad4b03b
1 parent b91d9cb
commit ad4b03b
Show file tree

Hide file tree

Showing 7 changed files with 368 additions and 65 deletions.
diff --git a/build-docker.sh b/build-docker.sh
@@ -1,8 +1,32 @@
 #!/bin/bash
 
-docker build -t ray-project/base-deps docker/base-deps
+while [[ $# -gt 0 ]]
+do
+key="$1"
+case $key in
+    --no-cache)
+    NO_CACHE="--no-cache"
+    ;;
+    --skip-examples)
+    SKIP_EXAMPLES=YES
+    ;;
+    *)
+    echo "Usage: build-docker.sh [ --no-cache ] [ --skip-examples ]"
+    exit 1
+esac
+shift
+done
 
-tar --exclude './docker' -c . > ./docker/deploy/ray.tar
+# Build base dependencies, allow caching
+docker build $NO_CACHE -t ray-project/base-deps docker/base-deps
+
+# Build the current Ray source
+git rev-parse HEAD > ./docker/deploy/git-rev
+git archive -o ./docker/deploy/ray.tar $(git rev-parse HEAD)
 docker build --no-cache -t ray-project/deploy docker/deploy
-rm ./docker/deploy/ray.tar
-docker build -t ray-project/examples docker/examples
+rm ./docker/deploy/ray.tar ./docker/deploy/git-rev
+
+
+if [ ! $SKIP_EXAMPLES ]; then
+    docker build $NO_CACHE -t ray-project/examples docker/examples
+fi
diff --git a/doc/source/conf.py b/doc/source/conf.py
@@ -43,10 +43,15 @@
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 
-# The suffix(es) of source filenames.
-# You can specify multiple suffix as a list of string:
-# source_suffix = ['.rst', '.md']
-source_suffix = '.rst'
+# The suffix of source filenames.
+from recommonmark.parser import CommonMarkParser
+
+# The suffix of source filenames.
+source_suffix = ['.rst', '.md']
+
+source_parsers = {
+   '.md': CommonMarkParser,
+}
 
 # The encoding of source files.
 #source_encoding = 'utf-8-sig'
@@ -64,9 +69,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '0.01'
+from ray import __version__ as version
 # The full version, including alpha/beta/rc tags.
-release = '0.01'
+release = version
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
@@ -297,13 +302,3 @@
 # sudo pip install recommonmark
 
 # see also http://searchvoidstar.tumblr.com/post/125486358368/making-pdfs-from-markdown-on-readthedocsorg-using
-
-# The suffix of source filenames.
-from recommonmark.parser import CommonMarkParser
-
-# The suffix of source filenames.
-source_suffix = ['.rst', '.md']
-
-source_parsers = {
-   '.md': CommonMarkParser,
-}
diff --git a/doc/source/index.rst b/doc/source/index.rst
@@ -6,7 +6,7 @@ Ray
 learning and reinforcement learning applications.*
 
 .. toctree::
-   :maxdepth: 0
+   :maxdepth: 1
    :caption: Installation
 
    install-on-ubuntu.md
@@ -15,7 +15,7 @@ learning and reinforcement learning applications.*
    installation-troubleshooting.md
 
 .. toctree::
-   :maxdepth: 0
+   :maxdepth: 1
    :caption: Examples
 
    example-hyperopt.md
@@ -24,7 +24,7 @@ learning and reinforcement learning applications.*
    using-ray-with-tensorflow.md
 
 .. toctree::
-   :maxdepth: 0
+   :maxdepth: 1
    :caption: Getting Started
 
    api.rst
@@ -43,3 +43,4 @@ learning and reinforcement learning applications.*
 
    using-ray-on-a-cluster.md
    using-ray-on-a-large-cluster.md
+   using-ray-and-docker-on-a-cluster.md
diff --git a/doc/source/install-on-docker.md b/doc/source/install-on-docker.md
@@ -1,10 +1,54 @@
 # Installation on Docker
 
-You can install Ray on any platform that runs Docker. We do not presently publish Docker images for Ray, but you can build them yourself using the Ray distribution. Using Docker can provide a reliable way to get up and running quickly.
+You can install Ray on any platform that runs Docker. We do not presently publish Docker images for Ray, but you can build them yourself using the Ray distribution.
+
+Using Docker can streamline the build process and provide a reliable way to get up and running quickly.
 
 ## Install Docker
 
-The Docker Platform release is available for Mac, Windows, and Linux platforms. Please download the appropriate version from the [Docker website](https://www.docker.com/products/overview#/install_the_platform).
+### Mac, Linux, Windows platforms
+
+The Docker Platform release is available for Mac, Windows, and Linux platforms. Please download the appropriate version from the [Docker website](https://www.docker.com/products/overview#/install_the_platform) and follow the corresponding installation instructions.
+Linux user may find these [alternate instructions](https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-16-04) helpful.
+
+### Docker installation on EC2 with Ubuntu
+
+The instructions below show in detail how to prepare an Amazon EC2 instance running Ubuntu 16.04 for use with Docker.
+
+Apply initialize the package repository and apply system updates:
+
+```
+sudo apt-get update
+sudo apt-get -y dist-upgrade
+```
+
+Install Docker and start the service:
+```
+sudo apt-get install -y docker.io
+sudo service docker start
+```
+
+Add the `ubuntu` user to the `docker` group to allow running Docker commands without sudo:
+```
+sudo usermod -a -G docker ubuntu
+```
+
+Initiate a new login to gain group permissions (alternatively, log out and log back in again):
+
+```
+exec sudo su -l ubuntu
+```
+
+Confirm that docker is running:
+
+```
+docker images
+```
+Should produce an empty table similar to the following:
+```
+REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
+```
+
 
 ## Clone the Ray repository
 
@@ -23,21 +67,50 @@ cd ray
 
 This script creates several Docker images:
 
- * The `ray-project/ray:deploy` image is a self-contained copy of code and binaries suitable for end users.
- * The `ray-project/ray:examples` adds additional libraries for running examples.
- * Ray developers who want to edit locally on the host filesystem should use the `ray-project/ray:devel` image, which allows local changes to be reflected immediately within the container.
+ * The `ray-project/deploy` image is a self-contained copy of code and binaries suitable for end users.
+ * The `ray-project/examples` adds additional libraries for running examples.
+ * The `ray-project/base-deps` image builds from Ubuntu Xenial and includes Anaconda and other basic dependencies and can serve as a starting point for developers.
+
+Review images by listing them:
+```
+$ docker images
+```
+
+Output should look something like the following:
+```
+REPOSITORY                          TAG                 IMAGE ID            CREATED             SIZE
+ray-project/examples                latest              7584bde65894        4 days ago          3.257 GB
+ray-project/deploy                  latest              970966166c71        4 days ago          2.899 GB
+ray-project/base-deps               latest              f45d66963151        4 days ago          2.649 GB
+ubuntu                              xenial              f49eec89601e        3 weeks ago         129.5 MB
+```
+
 
 ## Launch Ray in Docker
 
 Start out by launching the deployment container.
 
 ```
-docker run --shm-size=1024m -t -i ray-project/ray:deploy
+docker run --shm-size=<shm-size> -t -i ray-project/deploy
+```
+
+Replace `<shm-size>` with a limit appropriate for your system, for example `512M` or `2G`.
+The `-t` and `-i` options here are required to support interactive use of the container.
+
+**Note:** Ray requires a **large** amount of shared memory because each object
+store keeps all of its objects in shared memory, so the amount of shared memory
+will limit the size of the object store.
+
+You should now see a prompt that looks something like:
+
+```
+root@ebc78f68d100:/ray#
 ```
 
+
 ## Test if the installation succeeded
 
-To test if the installation was successful, try running some tests.
+To test if the installation was successful, try running some tests. Within the container shell enter the following commands:
 
 ```
 python test/runtest.py # This tests basic functionality.
@@ -52,55 +125,27 @@ Ray includes a Docker image that includes dependencies necessary for running som
 
 Launch the examples container.
 ```
-docker run --shm-size=1024m -t -i ray-project/ray:examples
+docker run --shm-size=1024m -t -i ray-project/examples
 ```
 
 ### Hyperparameter optimization
 
 
 ```
-cd ~/ray/examples/hyperopt/
+cd /ray/examples/hyperopt/
 python driver.py
 ```
 
-See the [Hyperparameter optimization documentation](../examples/hyperopt/README.md).
-
 ### Batch L-BFGS
 
 ```
-cd ~/ray/examples/lbfgs/
+cd /ray/examples/lbfgs/
 python driver.py
 ```
 
-See the [Batch L-BFGS documentation](../examples/lbfgs/README.md).
-
 ### Learning to play Pong
 
 ```
-cd ~/ray/examples/rl_pong/
+cd /ray/examples/rl_pong/
 python driver.py
 ```
-
-See the [Learning to play Pong documentation](../examples/rl_pong/README.md).
-
-
-## Developing with Docker (Experimental)
-
-These steps apply only to Ray developers who prefer to use editing tools on the host machine while building and running Ray within Docker. If you have previously been building locally we suggest that you start with a clean checkout before building with Ray's developer Docker container.
-
-You may see errors while running `setup.sh` on Mac OS X. If you have this problem please try re-running the script. Increasing the memory of Docker's VM (say to 8GB from the default 2GB) seems to help.
-
-
-Launch the developer container.
-
-```
-docker run -v $(pwd):/home/ray-user/ray --shm-size=1024m -t -i ray-project/ray:devel
-```
-
-Build Ray inside of the container.
-
-```
-cd ray
-./setup.sh
-./build.sh
-```