[Java] Build Java code with Bazel #4284

ruifangChen · 2019-03-06T07:34:03Z

After this PR, we can build Java code with Bazel as well.

To build Ray Java code, we should build Ray C++ code for Java as the following commands(under ray directory) firstly:

build.sh -l java

then

bazel build //java:all      # you can build any target in Java part

Then the Ray jar package will stand under generated directory.

Also, we can run Java tests by following java/test.sh

Related issue number

Fixes #4273

…nce/ray into ruifang.crf_stage

jovany-wang · 2019-03-06T08:10:47Z

Note that, after removing cmake build system, we should make sure we can support both Bazel and maven for building Java part.

raulchen

Thanks for submitting this! I left some quick comments. Will also try it out myself.

java/testng.xml

raulchen · 2019-03-06T07:49:25Z

java/test.sh

+  pushd $ROOT_DIR/../java/test
+  java -jar -Dray.home=$ROOT_DIR/../ -Djava.library.path=$ROOT_DIR/../bazel-bin:$ROOT_DIR/../bazel-bin/externl/plasma:$ROOT_DIR/../bazel-genfiles $ROOT_DIR/../bazel-bin/java/AllTests_deploy.jar $ROOT_DIR/../java/testng.xml
+
+  java -jar -Dray.home=$ROOT_DIR/../ -Dray.run-mode=SINGLE_PROCESS -Djava.library.path=$ROOT_DIR/../bazel-bin:$ROOT_DIR/../bazel-bin/externl/plasma:$ROOT_DIR/../bazel-genfiles $ROOT_DIR/../bazel-bin/java/AllTests_deploy.jar $ROOT_DIR/../java/testng.xml


It seems that you already link the binaries to /build/.... Why do we still need to overwrite -Djava.library.path?

BTW, I'm refining also this script at #4265.

also, it seems that we're not running checkstyle in bazel?

It seems that you already link the binaries to /build/.... Why do we still need to overwrite -Djava.library.path?

BTW, I'm refining also this script at #4265.

I forgot to delete the -Djava.library.path. I will fix it in this pr.

java/BUILD.bazel

guoyuhong · 2019-03-06T08:34:17Z

build.sh

-  cp -r $ROOT_DIR/bazel-genfiles/ray_pkg/ray $ROOT_DIR/python || true
+
+if [ "$RAY_BUILD_JAVA" == "YES" ]; then
+    bazel build //:ray_java_pkg -c opt


The arrow python package is now downloaded from AWS. It looks like arrow Java package still goes with the old way, say, downloading source files from GitHub and building locally. I remember there is a special fix for arrow to enable bazel. Is that special fix only for Python? @pcmoritz

We download the pyarrow files from AWS because it is compile differently from the official pyarrow package. This only concerns the python part (the official package is not compatible with the official tensorflow wheels).

raulchen · 2019-03-06T09:29:10Z

I run the test locally and got the following error:

ERROR: /Users/haochen/code/my_ray/java/BUILD.bazel:120:1: Executing genrule //java:generate_java_gcs_fbs failed (Exit 1)
Traceback (most recent call last):
  File "/private/var/tmp/_bazel_haochen/67c2e9ea7ec45db65d1c3b95cd173f21/execroot/com_github_riselab_ray/java/modify_generated_java_flatbuffers_files.py", line 55, in <module>
    add_package_declarations(root_path)
  File "/private/var/tmp/_bazel_haochen/67c2e9ea7ec45db65d1c3b95cd173f21/execroot/com_github_riselab_ray/java/modify_generated_java_flatbuffers_files.py", line 44, in add_package_declarations
    success = add_new_line(full_name, 2, PACKAGE_DECLARATION)
  File "/private/var/tmp/_bazel_haochen/67c2e9ea7ec45db65d1c3b95cd173f21/execroot/com_github_riselab_ray/java/modify_generated_java_flatbuffers_files.py", line 31, in add_new_line
    with open(file, "w") as file_handler:
PermissionError: [Errno 13] Permission denied: '/private/var/tmp/_bazel_haochen/67c2e9ea7ec45db65d1c3b95cd173f21/execroot/com_github_riselab_ray/java/runtime/src/main/java/org/ray/runtime/generated/ConfigTableData.java'

It seems that we need to chmod the generated files.

raulchen · 2019-03-06T09:31:47Z

Also, we need to delete this to make Java CI job run on bazel.

ray/.travis.yml

Line 28 in ff5e338

- RAY_USE_CMAKE=1

jovany-wang · 2019-03-06T09:33:58Z

I run the test locally and got the following error:

ERROR: /Users/haochen/code/my_ray/java/BUILD.bazel:120:1: Executing genrule //java:generate_java_gcs_fbs failed (Exit 1)
Traceback (most recent call last):
  File "/private/var/tmp/_bazel_haochen/67c2e9ea7ec45db65d1c3b95cd173f21/execroot/com_github_riselab_ray/java/modify_generated_java_flatbuffers_files.py", line 55, in <module>
    add_package_declarations(root_path)
  File "/private/var/tmp/_bazel_haochen/67c2e9ea7ec45db65d1c3b95cd173f21/execroot/com_github_riselab_ray/java/modify_generated_java_flatbuffers_files.py", line 44, in add_package_declarations
    success = add_new_line(full_name, 2, PACKAGE_DECLARATION)
  File "/private/var/tmp/_bazel_haochen/67c2e9ea7ec45db65d1c3b95cd173f21/execroot/com_github_riselab_ray/java/modify_generated_java_flatbuffers_files.py", line 31, in add_new_line
    with open(file, "w") as file_handler:
PermissionError: [Errno 13] Permission denied: '/private/var/tmp/_bazel_haochen/67c2e9ea7ec45db65d1c3b95cd173f21/execroot/com_github_riselab_ray/java/runtime/src/main/java/org/ray/runtime/generated/ConfigTableData.java'

It seems that we need to chmod the generated files.

The same to me. This could be reprod on mac.
I think we shouldn't use /private/var/tmp dir since the permission.

AmplabJenkins · 2019-03-06T10:24:43Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12594/
Test FAILed.

ruifangChen · 2019-03-06T11:05:04Z

I run the test locally and got the following error:

ERROR: /Users/haochen/code/my_ray/java/BUILD.bazel:120:1: Executing genrule //java:generate_java_gcs_fbs failed (Exit 1)
Traceback (most recent call last):
  File "/private/var/tmp/_bazel_haochen/67c2e9ea7ec45db65d1c3b95cd173f21/execroot/com_github_riselab_ray/java/modify_generated_java_flatbuffers_files.py", line 55, in <module>
    add_package_declarations(root_path)
  File "/private/var/tmp/_bazel_haochen/67c2e9ea7ec45db65d1c3b95cd173f21/execroot/com_github_riselab_ray/java/modify_generated_java_flatbuffers_files.py", line 44, in add_package_declarations
    success = add_new_line(full_name, 2, PACKAGE_DECLARATION)
  File "/private/var/tmp/_bazel_haochen/67c2e9ea7ec45db65d1c3b95cd173f21/execroot/com_github_riselab_ray/java/modify_generated_java_flatbuffers_files.py", line 31, in add_new_line
    with open(file, "w") as file_handler:
PermissionError: [Errno 13] Permission denied: '/private/var/tmp/_bazel_haochen/67c2e9ea7ec45db65d1c3b95cd173f21/execroot/com_github_riselab_ray/java/runtime/src/main/java/org/ray/runtime/generated/ConfigTableData.java'

It seems that we need to chmod the generated files.

I forgot to be compatible with the mac system, i will fix it in the pr.

delete the setting of -Djava.library.path

Modified according to the recommendations of raulchen Co-Authored-By: alegithub111 <ruif_chen111@126.com>

Modified according to the recommendations of raulchen

…nce/ray into ruifang.crf_stage

add the mvn ckeckstyle

Revert accidentally deleted code

Delete useless comments

raulchen · 2019-03-06T12:24:52Z

build.sh

+    ln -sf $ROOT_DIR/bazel-genfiles/redis-server $ROOT_DIR/build/src/ray/thirdparty/redis/src/redis-server
+    ln -sf $ROOT_DIR/bazel-bin/ray_redis_module.so $ROOT_DIR/build/src/ray/gcs/redis_module/libray_redis_module.so
+    ln -sf $ROOT_DIR/bazel-bin/external/plasma/plasma_store_server $ROOT_DIR/build/src/plasma/plasma_store_server
+    ln -sf $ROOT_DIR/bazel-bin/raylet $ROOT_DIR/build/src/ray/raylet/raylet


maybe just link these files to java/build (also need to change code accordingly). because after this PR, we don't need to support cmake.

if don't support cmake, we can delete these links and change the code.

raulchen · 2019-03-06T12:26:20Z

java/build.sh

@@ -1,3 +1,3 @@
 #!/usr/bin/env bash

-mvn clean install -Dmaven.test.skip
+bazel build -c opt //java:all


IIRC, this file isn't being used any more, we can just delete it.

leave the job to developers

if it's not used, the file should be deleted as part of this PR I'd say

jovany-wang · 2019-03-06T13:08:38Z

bazel/ray_java.bzl

@@ -0,0 +1,1577 @@
+# The following dependencies were calculated from:


If we don't auto-generate this file in this PR, let's leave a TODO comment here to take a note.

So do we truely need all these dependencies? It seems to me that this file could be slimmed down a lot, for example GRPC should not be needed to build this.

java/test.sh

AmplabJenkins · 2019-03-21T13:48:09Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/13143/
Test FAILed.

This reverts commit b504a22.

AmplabJenkins · 2019-03-21T14:18:53Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/13146/
Test FAILed.

AmplabJenkins · 2019-03-21T15:23:13Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/13153/
Test FAILed.

AmplabJenkins · 2019-03-21T19:12:55Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/13155/
Test FAILed.

AmplabJenkins · 2019-03-21T19:22:06Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/13158/
Test FAILed.

This reverts commit ba2b535.

AmplabJenkins · 2019-03-22T03:34:24Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/13176/
Test PASSed.

This reverts commit 101fbd1.

raulchen · 2019-03-22T04:54:46Z

Fixed some bugs, and CI is finally passing.

raulchen · 2019-03-22T04:55:37Z

@pcmoritz any luck with #4385?

pcmoritz · 2019-03-22T05:09:10Z

@raulchen: Awesome! #4385 is done now and just needs to be merged, see apache/arrow#4001. I'm also happy to merge this one first, depending on which one is ready first. One thing I still don't like about this is that java/third_party/workspace.bzl is checked in, is there no way to autogenerate that? Maybe it could be done before any other bazel command is run, so we can refer to it from the other Bazel files?

raulchen · 2019-03-22T05:14:42Z

@pcmoritz java/third_party/workspace.bzl is kind of tricky to deal with. It needs to be present before running any bazel command. Because it's referenced by the WORKSPACE file.
I searched around in github, all repos using the bazel-deps tool have to check in this file.
An alternative solution is using a script to generate it before running bazel. But it means that we'll need to run this script even if we only build Python. Because otherwise, the WORKSPACE file is invalid.
I'll do more research, and try to improve this in the next PR.

pcmoritz

Ok, sounds good to me!

raulchen · 2019-03-22T06:29:56Z

@pcmoritz Thanks. I'll merge this one first.

AmplabJenkins · 2019-03-22T07:15:40Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/13179/
Test PASSed.

ruifang.crf added 7 commits February 15, 2019 18:05

Refine JNI bazel script to make it suitable for more systems

6ff919e

Conflict resolution

a396961

java bazel config

ce314ff

Merge branch 'ruifang.crf_stage' of https://github.com/ant-tech-allia…

a1e7ea0

…nce/ray into ruifang.crf_stage

java bazel config

c39a75b

java bazel config

268e694

java bazel config

c63eb04

jovany-wang changed the title ~~java bazel config~~ [Java] Build Java code with Bazel Mar 6, 2019

robertnishihara assigned pcmoritz Mar 6, 2019

raulchen reviewed Mar 6, 2019

View reviewed changes

guoyuhong reviewed Mar 6, 2019

View reviewed changes

ruifangChen and others added 11 commits March 6, 2019 19:12

Update test.sh

c2bc0d8

delete the setting of -Djava.library.path

Update java/testng.xml

1b8a5a4

Modified according to the recommendations of raulchen Co-Authored-By: alegithub111 <ruif_chen111@126.com>

Update build.sh

40c0b62

Modified according to the recommendations of raulchen

Update test.sh

8eb90fa

Modified according to the recommendations of raulchen

Update build.sh

bded0ae

delete the variable RAY_USR_CMAKE

0263855

Merge branch 'ruifang.crf_stage' of https://github.com/ant-tech-allia…

a3ecbd7

…nce/ray into ruifang.crf_stage

Update test.sh

470518b

add the mvn ckeckstyle

Update .travis.yml

2b69448

Revert accidentally deleted code

Update .travis.yml

2aa6ceb

Update testng.xml

172e8cc

Delete useless comments

raulchen reviewed Mar 6, 2019

View reviewed changes

jovany-wang reviewed Mar 6, 2019

View reviewed changes

Supress output

cd0f583

Revert "fix generate_java_gcs_fbs"

6477aca

This reverts commit b504a22.

raulchen added 3 commits March 21, 2019 22:31

fix maven install

60073ae

pass in PATH

9d1fed2

debug

c71acfe

raulchen and others added 3 commits March 21, 2019 23:28

fix

330301b

fix

28c9169

modify problem: mvn command not found

ba2b535

raulchen added 2 commits March 22, 2019 08:53

Revert "modify problem: mvn command not found"

7943543

This reverts commit ba2b535.

skip test

101fbd1

raulchen added 2 commits March 22, 2019 12:05

Revert "skip test"

8c756d0

This reverts commit 101fbd1.

Fix loading native libraries

67b628f

raulchen closed this Mar 22, 2019

raulchen reopened this Mar 22, 2019

pcmoritz approved these changes Mar 22, 2019

View reviewed changes

raulchen merged commit 59d74d5 into ray-project:master Mar 22, 2019

raulchen deleted the ruifang.crf_stage branch March 22, 2019 06:30

robertnishihara mentioned this pull request Mar 22, 2019

Error initializing Ray #4193

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Java] Build Java code with Bazel #4284

[Java] Build Java code with Bazel #4284

ruifangChen commented Mar 6, 2019 •

edited by robertnishihara

Loading

jovany-wang commented Mar 6, 2019

raulchen left a comment

raulchen Mar 6, 2019

raulchen Mar 6, 2019

ruifangChen Mar 6, 2019

guoyuhong Mar 6, 2019 •

edited

Loading

pcmoritz Mar 6, 2019

raulchen commented Mar 6, 2019

raulchen commented Mar 6, 2019

jovany-wang commented Mar 6, 2019

AmplabJenkins commented Mar 6, 2019

ruifangChen commented Mar 6, 2019

raulchen Mar 6, 2019

ruifangChen Mar 6, 2019

raulchen Mar 6, 2019

ruifangChen Mar 6, 2019

pcmoritz Mar 7, 2019

jovany-wang Mar 6, 2019

ruifangChen Mar 6, 2019

pcmoritz Mar 7, 2019

AmplabJenkins commented Mar 21, 2019

AmplabJenkins commented Mar 21, 2019

AmplabJenkins commented Mar 21, 2019

AmplabJenkins commented Mar 21, 2019

AmplabJenkins commented Mar 21, 2019

AmplabJenkins commented Mar 22, 2019

raulchen commented Mar 22, 2019

raulchen commented Mar 22, 2019

pcmoritz commented Mar 22, 2019 •

edited

Loading

raulchen commented Mar 22, 2019

pcmoritz left a comment

raulchen commented Mar 22, 2019

AmplabJenkins commented Mar 22, 2019

		@@ -0,0 +1,1577 @@
		# The following dependencies were calculated from:

[Java] Build Java code with Bazel #4284

[Java] Build Java code with Bazel #4284

Conversation

ruifangChen commented Mar 6, 2019 • edited by robertnishihara Loading

Related issue number

jovany-wang commented Mar 6, 2019

raulchen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guoyuhong Mar 6, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raulchen commented Mar 6, 2019

raulchen commented Mar 6, 2019

jovany-wang commented Mar 6, 2019

AmplabJenkins commented Mar 6, 2019

ruifangChen commented Mar 6, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Mar 21, 2019

AmplabJenkins commented Mar 21, 2019

AmplabJenkins commented Mar 21, 2019

AmplabJenkins commented Mar 21, 2019

AmplabJenkins commented Mar 21, 2019

AmplabJenkins commented Mar 22, 2019

raulchen commented Mar 22, 2019

raulchen commented Mar 22, 2019

pcmoritz commented Mar 22, 2019 • edited Loading

raulchen commented Mar 22, 2019

pcmoritz left a comment

Choose a reason for hiding this comment

raulchen commented Mar 22, 2019

AmplabJenkins commented Mar 22, 2019

ruifangChen commented Mar 6, 2019 •

edited by robertnishihara

Loading

guoyuhong Mar 6, 2019 •

edited

Loading

pcmoritz commented Mar 22, 2019 •

edited

Loading