[JavaWorker] Enable java worker support

[salah-man]

---

This commit includes a tailored version of the Java worker implementation from Ant Financial.
The changes for build system, python module, src module and arrow are in other commits, this commit consists of the following modules:

• java/api: Ray API definition
• java/common: utilities
• java/hook: binary rewrite of the Java byte-code for remote execution
• java/runtime-common: common implementation of the runtime in worker
• java/runtime-dev: a pure-java mock implementation of the runtime for fast development
• java/runtime-native: a native implementation of the runtime
• java/test: various tests

Contributors for this work:
Guyang Song, Peng Cao, Senlin Zhu,Xiaoying Chu, Yiming Yu, Yujie Liu, Zhenyu Guo

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5474/
Test PASSed.

[robertnishihara]

#2095

[robertnishihara]

One thing that would be helpful is to include "quick start" instructions for getting a simple Ray application running from a brand new Ubuntu machine. To try write an application using Ray from Java, I had to do the following.

sudo apt install maven default-jdk

Then I created a project with mvn archetype:generate -DgroupId=com.mycompany.app -DartifactId=my-app -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false

Then I had to add the following to the pom.xml.

    <dependency>
      <groupId>org.ray</groupId>
      <artifactId>ray-api</artifactId>
      <version>1.0</version>
    </dependency>

    <dependency>
      <groupId>org.ray</groupId>
      <artifactId>ray-runtime-common</artifactId>
      <version>1.0</version>
    </dependency>

    <dependency>
      <groupId>org.ray</groupId>
      <artifactId>ray-runtime-native</artifactId>
      <version>1.0</version>
    </dependency>

    <dependency>
      <groupId>org.ray</groupId>
      <artifactId>ray-runtime-dev</artifactId>
      <version>1.0</version>
    </dependency>

    <dependency>
      <groupId>org.ray</groupId>
      <artifactId>ray-common</artifactId>
      <version>1.0</version>
    </dependency>

    <dependency>
      <groupId>org.ray</groupId>
      <artifactId>ray-hook</artifactId>
      <version>1.0</version>
    </dependency>

and also

  <properties>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
  </properties>

Then I had to add a bunch of imports like

import org.ray.api.Ray;
import org.ray.api.RayRemote;
import org.ray.api.RayActor;
import org.ray.api.RayList;
import org.ray.api.RayObject;
import org.ray.api.WaitResult;
import org.ray.util.logger.RayLog;

to the application.

Then I had to duplicate ray/java/ray.config.ini and add %CONFIG_FILE_DIR%/../../my-app/target/my-app-1.0-SNAPSHOT.jar = under the [ray.java.path.classes.package] header.

Then I compiled with mvn package.

Then I had to do export RAY_CONFIG=/path/to/ray.config.ini.

Then I compiled with java -Djava.library.path=/home/ubuntu/ray/build/src/plasma/:/home/ubuntu/ray/build/src/local_scheduler/ -cp target/my-app-1.0-SNAPSHOT.jar:/home/ubuntu/ray/java/test/lib/* com.mycompany.app.App

I think we can come up with much simpler instructions (e.g., we can provide an example so the user doesn't have to do mvn archetype:generate and so the pom.xml is already in place.

[robertnishihara]

I had to replace this with ../build.sh -l java to make it work. With sh I got ../build.sh: 10: ../build.sh: Syntax error: "(" unexpected.

[salah-man]

I will fix it soon, thanks.

[robertnishihara]

It'd be preferable to convert the documentation to RST (instead of markdown) for consistency with the rest of the documentation.

[salah-man]

It's ok to me, i could try to change the format from markdown to RST.

[salah-man]

Thanks for your time @robertnishihara.
First, the class under test package actually the sample or demo case besides the test case.
Second, i could add a module like example which has the simple hello world case.

[salah-man]

Excuse me, but after all these pr(inclue the pr in Arrow) merged, we still need to start a final pr to let the java worker work well, which need to update the version of arrow, add the java test to the Ray ci, and some adaptation work. And the code and test is ready at my local, but i could not submit the pr when the pr currently submited not merged, because i could not have the right base code. How about add the example module in the final pr, and of couse i could change the Readme.md to RST format in this pr. Thank you very much @robertnishihara .

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5630/
Test PASSed.

[robertnishihara]

@salah-man sounds good, I think it is fine to add the example in a subsequent PR.

[pcmoritz]

@salah-man This looks good, but before we can merge it, can you please add an entry in the Travis build matrix to test it? For a start it's fine to just add some simple tests to test the basic functionality of the API. I think it also needs to use apache/arrow#2065 to work.

[salah-man]

Hi @pcmoritz you are right, and i still have the final PR at my local, which is for the version update of arrow, add java test case to the Travis ci and some adaption work, and the code is ready. And because the final PR is based on the current PR, so i do not submit before this PR merged. So whether i should submit the final PR into the current PR with different commit or submit another independent PR?

[pcmoritz]

Can you submit the final PR so we can have a look at it? It doesn't matter that it is based on this PR for now :)

[salah-man]

@pcmoritz Ok, i will rebase the master branch from current branch and cherry-pick the adaption, and i could the submit the final PR into current PR.

[pcmoritz]

Just submitting the PR so we can see the diff to this one would also be ok, then we can decide if we want to put them together or merge them separately.

[salah-man]

OK i see, i could submit a PR based on the branch of java_worker_mr3, and i need a little time to clean the code because i notice that the master branch have do some work the same as me (some thing like 225608e). And i will finish code clean and test as soon as possible. Thank you very much @pcmoritz

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5637/
Test FAILed.

[salah-man]

Hi, here comes the final PR, which compare the code branch of java_worker_mr3, Thanks.
antgroup#15

[pcmoritz]

Thanks, that looks good! Let's add the changes on top of this PR and merge the changes together, so we have a fully functional PR integrated.

[salah-man]

Ok, i have already add the changes on top of this PR.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5638/
Test FAILed.

[pcmoritz]

Jenkins retest this please

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5640/
Test PASSed.

[pcmoritz]

+1, thanks a lot for getting this merged!

[robertnishihara]

Awesome, thanks for contributing this!

[salah-man]

Thank you very much, your comments are really helpful and professional, and look forward to the next cooperation @robertnishihara @pcmoritz. And thank you for your time again.