Change ray.worker.cleanup -> ray.shutdown and improve API documentation. (ray-project#2374)

* Change ray.worker.cleanup -> ray.shutdown and improve API documentation.

* Deprecate ray.worker.cleanup() gracefully.

* Fix linting

Author: robertnishihara

doc/requirements-doc.txt

@@ -11,5 +11,6 @@ psutil
 recommonmark
 redis
 sphinx
+sphinx-click
 sphinx_rtd_theme
 pandas

doc/source/api.rst

@@ -1,284 +1,49 @@
 The Ray API
 ===========
 
-Starting Ray
-------------
-
-There are two main ways in which Ray can be used. First, you can start all of
-the relevant Ray processes and shut them all down within the scope of a single
-script. Second, you can connect to and use an existing Ray cluster.
-
-Starting and stopping a cluster within a script
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-One use case is to start all of the relevant Ray processes when you call
-``ray.init`` and shut them down when the script exits. These processes include
-local and global schedulers, an object store and an object manager, a redis
-server, and more.
-
-**Note:** this approach is limited to a single machine.
-
-This can be done as follows.
-
-.. code-block:: python
-
-  ray.init()
-
-If there are GPUs available on the machine, you should specify this with the
-``num_gpus`` argument. Similarly, you can also specify the number of CPUs with
-``num_cpus``.
-
-.. code-block:: python
-
-  ray.init(num_cpus=20, num_gpus=2)
-
-By default, Ray will use ``psutil.cpu_count()`` to determine the number of CPUs.
-Ray will also attempt to automatically determine the number of GPUs.
-
-Instead of thinking about the number of "worker" processes on each node, we
-prefer to think in terms of the quantities of CPU and GPU resources on each
-node and to provide the illusion of an infinite pool of workers. Tasks will be
-assigned to workers based on the availability of resources so as to avoid
-contention and not based on the number of available worker processes.
-
-Connecting to an existing cluster
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Once a Ray cluster has been started, the only thing you need in order to connect
-to it is the address of the Redis server in the cluster. In this case, your
-script will not start up or shut down any processes. The cluster and all of its
-processes may be shared between multiple scripts and multiple users. To do this,
-you simply need to know the address of the cluster's Redis server. This can be
-done with a command like the following.
-
-.. code-block:: python
-
-  ray.init(redis_address="12.345.67.89:6379")
-
-In this case, you cannot specify ``num_cpus`` or ``num_gpus`` in ``ray.init``
-because that information is passed into the cluster when the cluster is started,
-not when your script is started.
-
-View the instructions for how to `start a Ray cluster`_ on multiple nodes.
-
-.. _`start a Ray cluster`: http://ray.readthedocs.io/en/latest/using-ray-on-a-cluster.html
-
 .. autofunction:: ray.init
 
-Defining remote functions
--------------------------
-
-Remote functions are used to create tasks. To define a remote function, the
-``@ray.remote`` decorator is placed over the function definition.
-
-The function can then be invoked with ``f.remote``. Invoking the function
-creates a **task** which will be scheduled on and executed by some worker
-process in the Ray cluster. The call will return an **object ID** (essentially a
-future) representing the eventual return value of the task. Anyone with the
-object ID can retrieve its value, regardless of where the task was executed (see
-`Getting values from object IDs`_).
-
-When a task executes, its outputs will be serialized into a string of bytes and
-stored in the object store.
-
-Note that arguments to remote functions can be values or object IDs.
-
-.. code-block:: python
-
-  @ray.remote
-  def f(x):
-      return x + 1
-
-  x_id = f.remote(0)
-  ray.get(x_id)  # 1
-
-  y_id = f.remote(x_id)
-  ray.get(y_id)  # 2
-
-If you want a remote function to return multiple object IDs, you can do that by
-passing the ``num_return_vals`` argument into the remote decorator.
-
-.. code-block:: python
-
-  @ray.remote(num_return_vals=2)
-  def f():
-      return 1, 2
-
-  x_id, y_id = f.remote()
-  ray.get(x_id)  # 1
-  ray.get(y_id)  # 2
-
 .. autofunction:: ray.remote
 
-Getting values from object IDs
-------------------------------
-
-Object IDs can be converted into objects by calling ``ray.get`` on the object
-ID. Note that ``ray.get`` accepts either a single object ID or a list of object
-IDs.
-
-.. code-block:: python
-
-  @ray.remote
-  def f():
-      return {'key1': ['value']}
-
-  # Get one object ID.
-  ray.get(f.remote())  # {'key1': ['value']}
-
-  # Get a list of object IDs.
-  ray.get([f.remote() for _ in range(2)])  # [{'key1': ['value']}, {'key1': ['value']}]
-
-Numpy arrays
-~~~~~~~~~~~~
-
-Numpy arrays are handled more efficiently than other data types, so **use numpy
-arrays whenever possible**.
-
-Any numpy arrays that are part of the serialized object will not be copied out
-of the object store. They will remain in the object store and the resulting
-deserialized object will simply have a pointer to the relevant place in the
-object store's memory.
-
-Since objects in the object store are immutable, this means that if you want to
-mutate a numpy array that was returned by a remote function, you will have to
-first copy it.
-
 .. autofunction:: ray.get
 
-Putting objects in the object store
------------------------------------
-
-The primary way that objects are placed in the object store is by being returned
-by a task. However, it is also possible to directly place objects in the object
-store using ``ray.put``.
-
-.. code-block:: python
-
-  x_id = ray.put(1)
-  ray.get(x_id)  # 1
-
-The main reason to use ``ray.put`` is that you want to pass the same large
-object into a number of tasks. By first doing ``ray.put`` and then passing the
-resulting object ID into each of the tasks, the large object is copied into the
-object store only once, whereas when we directly pass the object in, it is
-copied multiple times.
-
-.. code-block:: python
-
-  import numpy as np
-
-  @ray.remote
-  def f(x):
-      pass
-
-  x = np.zeros(10 ** 6)
-
-  # Alternative 1: Here, x is copied into the object store 10 times.
-  [f.remote(x) for _ in range(10)]
-
-  # Alternative 2: Here, x is copied into the object store once.
-  x_id = ray.put(x)
-  [f.remote(x_id) for _ in range(10)]
-
-Note that ``ray.put`` is called under the hood in a couple situations.
-
-- It is called on the values returned by a task.
-- It is called on the arguments to a task, unless the arguments are Python
-  primitives like integers or short strings, lists, tuples, or dictionaries.
+.. autofunction:: ray.wait
 
 .. autofunction:: ray.put
 
-Waiting for a subset of tasks to finish
----------------------------------------
-
-It is often desirable to adapt the computation being done based on when
-different tasks finish. For example, if a bunch of tasks each take a variable
-length of time, and their results can be processed in any order, then it makes
-sense to simply process the results in the order that they finish. In other
-settings, it makes sense to discard straggler tasks whose results may not be
-needed.
-
-To do this, we introduce the ``ray.wait`` primitive, which takes a list of
-object IDs and returns when a subset of them are available. By default it blocks
-until a single object is available, but the ``num_returns`` value can be
-specified to wait for a different number. If a ``timeout`` argument is passed
-in, it will block for at most that many milliseconds and may return a list with
-fewer than ``num_returns`` elements.
-
-The ``ray.wait`` function returns two lists. The first list is a list of object
-IDs of available objects (of length at most ``num_returns``), and the second
-list is a list of the remaining object IDs, so the combination of these two
-lists is equal to the list passed in to ``ray.wait`` (up to ordering).
-
-.. code-block:: python
-
-  import time
-  import numpy as np
+.. autofunction:: ray.get_gpu_ids
 
-  @ray.remote
-  def f(n):
-      time.sleep(n)
-      return n
-
-  # Start 3 tasks with different durations.
-  results = [f.remote(i) for i in range(3)]
-  # Block until 2 of them have finished.
-  ready_ids, remaining_ids = ray.wait(results, num_returns=2)
-
-  # Start 5 tasks with different durations.
-  results = [f.remote(i) for i in range(5)]
-  # Block until 4 of them have finished or 2.5 seconds pass.
-  ready_ids, remaining_ids = ray.wait(results, num_returns=4, timeout=2500)
-
-It is easy to use this construct to create an infinite loop in which multiple
-tasks are executing, and whenever one task finishes, a new one is launched.
-
-.. code-block:: python
-
-  @ray.remote
-  def f():
-      return 1
-
-  # Start 5 tasks.
-  remaining_ids = [f.remote() for i in range(5)]
-  # Whenever one task finishes, start a new one.
-  for _ in range(100):
-      ready_ids, remaining_ids = ray.wait(remaining_ids)
-      # Get the available object and do something with it.
-      print(ray.get(ready_ids))
-      # Start a new task.
-      remaining_ids.append(f.remote())
-
-.. autofunction:: ray.wait
+.. autofunction:: ray.get_resource_ids
 
-Viewing errors
---------------
+.. autofunction:: ray.get_webui_url
 
-Keeping track of errors that occur in different processes throughout a cluster
-can be challenging. There are a couple mechanisms to help with this.
+.. autofunction:: ray.shutdown
 
-1. If a task throws an exception, that exception will be printed in the
-   background of the driver process.
+.. autofunction:: ray.register_custom_serializer
 
-2. If ``ray.get`` is called on an object ID whose parent task threw an exception
-   before creating the object, the exception will be re-raised by ``ray.get``.
+.. autofunction:: ray.profile
 
-The errors will also be accumulated in Redis and can be accessed with
-``ray.error_info``. Normally, you shouldn't need to do this, but it is possible.
+.. autofunction:: ray.method
 
-.. code-block:: python
+The Ray Command Line API
+------------------------
 
-  @ray.remote
-  def f():
-      raise Exception("This task failed!!")
+.. click:: ray.scripts.scripts:start
+   :prog: ray start
+   :show-nested:
 
-  f.remote()  # An error message will be printed in the background.
+.. click:: ray.scripts.scripts:stop
+   :prog: ray stop
+   :show-nested:
 
-  # Wait for the error to propagate to Redis.
-  import time
-  time.sleep(1)
+.. click:: ray.scripts.scripts:create_or_update
+   :prog: ray create_or_update
+   :show-nested:
 
-  ray.error_info()  # This returns a list containing the error message.
+.. click:: ray.scripts.scripts:teardown
+   :prog: ray teardown
+   :show-nested:
 
-.. autofunction:: ray.error_info
+.. click:: ray.scripts.scripts:get_head_ip
+   :prog: ray get_head_ip
+   :show-nested:

doc/source/conf.py

@@ -72,6 +72,7 @@
 extensions = [
     'sphinx.ext.autodoc',
     'sphinx.ext.napoleon',
+    'sphinx_click.ext',
 ]
 
 # Add any paths that contain templates here, relative to this directory.

python/ray/__init__.py

@@ -50,8 +50,8 @@
 from ray.worker import (error_info, init, connect, disconnect, get, put, wait,
                         remote, profile, flush_profile_data, get_gpu_ids,
                         get_resource_ids, get_webui_url,
-                        register_custom_serializer)  # noqa: E402
-from ray.worker import (SCRIPT_MODE, WORKER_MODE, PYTHON_MODE,
+                        register_custom_serializer, shutdown)  # noqa: E402
+from ray.worker import (SCRIPT_MODE, WORKER_MODE, LOCAL_MODE,
                         SILENT_MODE)  # noqa: E402
 from ray.worker import global_state  # noqa: E402
 # We import ray.actor because some code is run in actor.py which initializes
@@ -67,8 +67,9 @@
     "error_info", "init", "connect", "disconnect", "get", "put", "wait",
     "remote", "profile", "flush_profile_data", "actor", "method",
     "get_gpu_ids", "get_resource_ids", "get_webui_url",
-    "register_custom_serializer", "SCRIPT_MODE", "WORKER_MODE", "PYTHON_MODE",
-    "SILENT_MODE", "global_state", "ObjectID", "_config", "__version__"
+    "register_custom_serializer", "shutdown", "SCRIPT_MODE", "WORKER_MODE",
+    "LOCAL_MODE", "SILENT_MODE", "global_state", "ObjectID", "_config",
+    "__version__"
 ]
 
 import ctypes  # noqa: E402