Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Serve] Java Serve improvement #42

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
serve build and run support Java file.
Signed-off-by: chuhan.ly <chuhan.ly@antgroup.com>
  • Loading branch information
chuhan.ly committed Sep 29, 2023
commit d3ede4924805be3833221cbe23475b0064f3079f
160 changes: 130 additions & 30 deletions reps/2023-08-18-serve-java-dag-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,47 +123,144 @@ deployment = serve.deployment('io.ray.serve.ExampleDeployment', name='my_deploym

```
### Deploying through the Config File
Using a config file, we can deploy the Serve application through the CLI. For Java deployment, we can also describe it in a Python file and generate the corresponding deployment configuration in the config file. Let's take the example of the text.py file:
```python
from ray import serve
from ray.serve.generated.serve_pb2 import JAVA
Based on the mentioned API, we can deploy a Java code file that orchestrates an application using the `serve run` command. For example, consider the following Text.java file:
```java
import io.ray.serve.api.Serve;
import io.ray.serve.deployment.Application;
import io.ray.serve.handle.DeploymentHandle;

public class Text {

@serve.deployment
class Hello:
def __call__(self) -> str:
return "Hello"
public static class Hello {
public String call() {
return "Hello";
}
}

public static class World {
public String call() {
return " world!";
}
}

world_java = serve.deployment('io.ray.serve.World', language=JAVA)
public static class Ingress {
private DeploymentHandle helloHandle;
private DeploymentHandle worldHandle;

public Ingress(DeploymentHandle helloHandle, DeploymentHandle worldHandle) {
this.helloHandle = helloHandle;
this.worldHandle = worldHandle;
}

@serve.deployment
class Ingress:
def __init__(self, hello_handle, world_handle):
self._hello_handle = hello_handle.options(
use_new_handle_api=True,
)
self._world_handle = world_handle.options(
use_new_handle_api=True,
)
public String call() {
return (String) helloHandle.remote().result() + worldHandle.remote().result();
}
}

public static Application app() {
Application hello = Serve.deployment().setDeploymentDef(Hello.class.getName()).bind();
Application world = Serve.deployment().setDeploymentDef(World.class.getName()).bind();

Application app =
Serve.deployment().setDeploymentDef(Ingress.class.getName()).bind(hello, world);
return app;
}
}

```
This code orchestrates an application within a static method named `app`. The CLI command for its deployment is as follows:
```shell
$ serve run java:io.ray.serve.repdemo.Text:app
```

> It should be noted that `java` is mentioned here to represent the programming language used for orchestrating the application. Currently, there is no clear plan on how to expand this parameter in the serve run command. One direct approach that comes to mind is to concatenate it before the `import_path`.
liuyang-my marked this conversation as resolved.
Show resolved Hide resolved

Additionally, similar to the Python `app_builder`, a Java application also supports custom parameters. For example:
```java
import io.ray.serve.api.Serve;
import io.ray.serve.deployment.Application;
import java.util.Map;

public class Hello {

public static class HelloWorld {
private String message;

public HelloWorld(String message) {
this.message = message;
}

public String call() {
return message;
}
}

async def __call__(self) -> str:
hello_response = self._hello_handle.remote()
world_response = self._world_handle.remote()
return (await hello_response) + (await world_response)
public static Application appBuilder(Map<String, String> args) {
return Serve.deployment()
.setDeploymentDef(HelloWorld.class.getName())
.bind(args.get("message"));
}
}

```

hello = Hello.bind()
world = world_java.bind()
The `appBuilder` method takes a `Map` as input parameter, from which users can retrieve the required `message` parameter. The syntax for deployment is as follows:

app = Ingress.bind(hello, world)

```shell
$ serve run java:io.ray.serve.repdemo.Hello:appBuilder message="Hello from CLI"
```
In this code, we define the Java Deployment `World` using the Python API and then bind it with a Python Deployment `Hello` into the `Ingress` to form an app. By using the `serve build` command, the Serve config file can be generated.

Furthermore, it is worth mentioning that `appBuilder` also supports user-defined input parameter of custom types, as long as the type includes the specified attributes. For example:

```java
import io.ray.serve.api.Serve;
import io.ray.serve.deployment.Application;
import java.util.Map;

public class Hello {

public static class HelloWorldArgs {
private String message;

public String getMessage() {
return message;
}

public void setMessage(String message) {
this.message = message;
}
}

public static class HelloWorld {
private String message;

public HelloWorld(String message) {
this.message = message;
}

public String call() {
return message;
}
}

public static Application typedAppBuilder(HelloWorldArgs args) {
return Serve.deployment().setDeploymentDef(HelloWorld.class.getName()).bind(args.getMessage());
}
}

```

```shell
$ serve run java:io.ray.serve.repdemo.Hello:typedAppBuilder message="Hello from CLI"
```

For the aforementioned `Text.java` file, we can generate the corresponding Serve config file using the `serve build` command:

```shell
$ serve build text:app -o serve_config.yaml
$ serve build io.ray.serve.repdemo.Text:app -o serve_config.yaml
```

The generated config file looks like this:
```yaml
proxy_location: EveryNode
Expand All @@ -177,17 +274,20 @@ grpc_options:
grpc_servicer_functions: []

applications:
- name: app1
- name: app
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this can take args: parameter as well (will be passed in the same way that it is via command line)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, the example of adding args in the config file has been supplemented.

route_prefix: /
import_path: text:app
import_path: io.ray.serve.repdemo.Text:app
language: java
runtime_env: {}
deployments:
- name: Hello
- name: World
- name: Ingress

```
By using this serve config file, the Application can be deployed through the "serve run" command:

The `serve run` command allows direct specification of this config file for deploying the application:

```shell
$ serve run serve_config.yaml
```
Expand Down