[Serve] Add experimental pipeline docs #20292

simon-mo · 2021-11-12T17:04:11Z

Why are these changes needed?

This PR adds documentation for the ray.serve.pipeline module.

Related issue number

Closes #19478

Checks

I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

simon-mo · 2021-11-13T01:01:45Z

python/ray/serve/pipeline/executor.py

@@ -24,7 +24,8 @@ async def call_async(self, input_arg: Tuple[Any]) -> Union[Any, ObjectRef]:
 class LocalExecutor(Executor):
    """Executor that runs code in-process."""

-    def __init__(self, serialized_callable_factory: bytes, config: StepConfig):
+    def __init__(self, serialized_callable_factory: bytes,
+                 _config: StepConfig):


unused variable, underscore it.

simon-mo · 2021-11-13T01:02:06Z

python/ray/serve/pipeline/node.py

@@ -26,11 +25,8 @@ class Pipeline:
    def __init__(self, entry_node: PipelineNode):
        self._entry_node = entry_node

-    def deploy(self):


best error is attribute doesn't exist :D.

simon-mo · 2021-11-13T01:02:22Z

@edoakes ready for review

edoakes · 2021-11-13T01:25:08Z

@richardliaw help review too please?

richardliaw · 2021-11-13T01:53:52Z

gave feedback offline.

edoakes

Awesome start!

edoakes · 2021-11-14T03:09:24Z

doc/source/serve/ml-models.rst

@@ -44,6 +44,10 @@ dive.
 Model Composition
 =================

+.. note::
+  Serve recently added experimental first-class model composition API. Please


Suggested change

Serve recently added experimental first-class model composition API. Please

Serve recently added an experimental first-class API for model composition (pipelines). Please

please try to be careful about grammar in documentation

edoakes · 2021-11-14T03:09:32Z

doc/source/serve/ml-models.rst

@@ -44,6 +44,10 @@ dive.
 Model Composition
 =================

+.. note::
+  Serve recently added experimental first-class model composition API. Please
+  take a look at :ref:`Pipeline API <serve-pipeline-api>` and try it out!


Suggested change

take a look at :ref:`Pipeline API <serve-pipeline-api>` and try it out!

take a look at the :ref:`Pipeline API <serve-pipeline-api>` and try it out!

Please add a call to action for people to file issues or post on forum w/ questions.

edoakes · 2021-11-14T03:09:56Z

doc/source/serve/pipeline.rst

+- understand the experimental pipeline API
+- build on top of the API to construct your multi-model inference pipelines.


inconsistent period usage

edoakes · 2021-11-14T03:10:11Z

doc/source/serve/pipeline.rst

+
+.. note::
+    This API is experimental and the API is subject to change.
+    We are actively looking for feedback via the Ray Slack or Github issues.


add links for these

Suggested change

We are actively looking for feedback via the Ray Slack or Github issues.

We are actively looking for feedback via the Ray Slack or Github issues.

edoakes · 2021-11-14T03:11:03Z

doc/source/serve/pipeline.rst

+Serve Pipeline is a standalone package that can be used without Ray Serve. 
+However, the expected usage is to use it inside your Serve deployment.


kind of a weird thing to open with. I'd start with the problem statement: This package is mean to make it seamless to develop and deploy multi-model pipelines. Add a sentence or two about what the advantages are (all in code, good perf)

edoakes · 2021-11-14T03:13:25Z

doc/source/serve/pipeline.rst

+For classes, you need to instantiate them with init args first, then pass in their
+upstream nodes. 


Suggested change

For classes, you need to instantiate them with init args first, then pass in their

upstream nodes.

For classes, you need to instantiate them with init args first, then pass in their

upstream nodes. This allows you to have the same code definition but pass different arguments, like URIs for model weights.

we should add an example for that

edoakes · 2021-11-14T03:13:44Z

doc/source/serve/pipeline.rst

+
+.. autofunction:: ray.serve.pipeline.step
+
+Here is an example pipeline that uses actors instead of local execution mode.


say that local execution mode is useful for testing?

edoakes · 2021-11-14T03:13:52Z

doc/source/serve/pipeline.rst

+----------------
+
+In this section, we show how to implement a two stage pipeline that's common
+for computer vision task. For workload like image classification, the preprocessing


Suggested change

for computer vision task. For workload like image classification, the preprocessing

for computer vision tasks. For workloads like image classification, the preprocessing

edoakes · 2021-11-14T03:14:02Z

doc/source/serve/pipeline.rst

+In this section, we show how to implement a two stage pipeline that's common
+for computer vision task. For workload like image classification, the preprocessing
+steps are CPU bounded and hard to parallelize. The actual inference steps can run
+on GPU and batched. They are often split up into separate stages and scaled separately


why do I want to batch it?

edoakes · 2021-11-14T03:15:47Z

doc/source/serve/pipeline.rst

+    :end-before: __preprocessing_pipeline_example_end__
+
+
+Ensemble Example


can you add a section after this listing more potential use cases?

Also, can we add a simple example of "combining business logic + ML"? This could be just calling out that you can do anything in the "combine output" step

edoakes · 2021-11-14T03:20:40Z

@simon-mo can you also add a pointer to this on the index page? Can say something like "Serve is great for model composition -- check out our new pipeline API!"

…ne-docs

edoakes · 2021-11-16T20:22:43Z

doc/source/serve/index.rst

+Ray Serve enables seamless multi-models inference pipeline (also known as model composition). You can
+write your inference pipeline all in code and integrate business logic with ML.


add link to pipelines here?

waleedkadous · 2021-11-17T21:23:05Z

python/ray/serve/pipeline/tests/test_pipeline.py

@@ -145,5 +148,18 @@ def combiner(greeting1: str, greeting2: str):
    assert greeter.call("Teddy") == "Howdy Teddy!|How's it hanging, Teddy?"


+def test_cleanup_and_destory(shared_ray_instance):


destory --> destroy

empty merge commit

simon-mo added 2 commits November 11, 2021 14:30

wip

9f29a7a

[Serve] Add docs for experimental pipeline

7fc4983

simon-mo assigned edoakes Nov 12, 2021

simon-mo changed the title ~~[Serve] Add experimental pipeline docs~~ [Serve] [WIP] Add experimental pipeline docs Nov 12, 2021

simon-mo added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Nov 12, 2021

simon-mo added 3 commits November 12, 2021 14:50

ensemble snippet

af362fe

polish

fd8ed5f

Add doc page

f1478d2

simon-mo changed the title ~~[Serve] [WIP] Add experimental pipeline docs~~ [Serve] Add experimental pipeline docs Nov 13, 2021

simon-mo marked this pull request as ready for review November 13, 2021 01:01

simon-mo removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Nov 13, 2021

simon-mo requested a review from edoakes November 13, 2021 01:01

simon-mo commented Nov 13, 2021

View reviewed changes

edoakes assigned richardliaw Nov 13, 2021

edoakes reviewed Nov 14, 2021

View reviewed changes

simon-mo added 5 commits November 15, 2021 11:15

Merge branch 'master' of github.com:ray-project/ray into serve/pipeli…

b6d9d0b

…ne-docs

lint

be5650c

Comments

d041242

fix lint

da4aa19

lint

09ac2ca

simon-mo requested a review from edoakes November 16, 2021 20:11

edoakes approved these changes Nov 16, 2021

View reviewed changes

simon-mo added 3 commits November 16, 2021 12:57

lint again

01daeea

lint again

e683755

link

a37aee4

fix

615572b

simon-mo merged commit 5fccad4 into ray-project:master Nov 17, 2021

fishbone pushed a commit that referenced this pull request Nov 17, 2021

[Serve] Add experimental pipeline docs (#20292)

8f937f4

waleedkadous reviewed Nov 17, 2021

View reviewed changes

wuisawesome pushed a commit that referenced this pull request Nov 20, 2021

[Serve] Add experimental pipeline docs (#20292)

1cf8f68

wuisawesome pushed a commit that referenced this pull request Nov 21, 2021

[Serve] Add experimental pipeline docs (#20292)

0dc6318

jiaodong pushed a commit to jiaodong/ray that referenced this pull request Dec 8, 2021

[Serve] Add experimental pipeline docs (ray-project#20292)

5ef603a

empty merge commit

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Serve] Add experimental pipeline docs #20292

[Serve] Add experimental pipeline docs #20292

simon-mo commented Nov 12, 2021 •

edited

Loading

simon-mo Nov 13, 2021

simon-mo Nov 13, 2021

simon-mo commented Nov 13, 2021

edoakes commented Nov 13, 2021

richardliaw commented Nov 13, 2021

edoakes left a comment

edoakes Nov 14, 2021

edoakes Nov 14, 2021

edoakes Nov 14, 2021

edoakes Nov 14, 2021

edoakes Nov 14, 2021

edoakes Nov 14, 2021

edoakes Nov 14, 2021

edoakes Nov 14, 2021

edoakes Nov 14, 2021

edoakes Nov 14, 2021

edoakes Nov 14, 2021

edoakes Nov 14, 2021

edoakes commented Nov 14, 2021

edoakes Nov 16, 2021

waleedkadous Nov 17, 2021

	Serve recently added experimental first-class model composition API. Please
	Serve recently added an experimental first-class API for model composition (pipelines). Please

	take a look at :ref:`Pipeline API <serve-pipeline-api>` and try it out!
	take a look at the :ref:`Pipeline API <serve-pipeline-api>` and try it out!

		- understand the experimental pipeline API
		- build on top of the API to construct your multi-model inference pipelines.

	We are actively looking for feedback via the Ray Slack or Github issues.
	We are actively looking for feedback via the Ray Slack or Github issues.

		Serve Pipeline is a standalone package that can be used without Ray Serve.
		However, the expected usage is to use it inside your Serve deployment.

		For classes, you need to instantiate them with init args first, then pass in their
		upstream nodes.


		.. autofunction:: ray.serve.pipeline.step

		Here is an example pipeline that uses actors instead of local execution mode.

	for computer vision task. For workload like image classification, the preprocessing
	for computer vision tasks. For workloads like image classification, the preprocessing

		:end-before: __preprocessing_pipeline_example_end__


		Ensemble Example

		Ray Serve enables seamless multi-models inference pipeline (also known as model composition). You can
		write your inference pipeline all in code and integrate business logic with ML.

		@@ -145,5 +148,18 @@ def combiner(greeting1: str, greeting2: str):
		assert greeter.call("Teddy") == "Howdy Teddy!\|How's it hanging, Teddy?"


		def test_cleanup_and_destory(shared_ray_instance):

[Serve] Add experimental pipeline docs #20292

[Serve] Add experimental pipeline docs #20292

Conversation

simon-mo commented Nov 12, 2021 • edited Loading

Why are these changes needed?

Related issue number

Checks

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simon-mo commented Nov 13, 2021

edoakes commented Nov 13, 2021

richardliaw commented Nov 13, 2021

edoakes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edoakes commented Nov 14, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simon-mo commented Nov 12, 2021 •

edited

Loading