Synchronize arrow version and put changes upstream #4385

pcmoritz · 2019-03-15T22:00:10Z

The goal of this PR is to make sure the version of arrow linked into Ray via Bazel and the python wheels are at the same version and synchronized with the latest arrow master.

Merge ARROW-4905: [C++][Plasma] Remove dlmalloc symbols from client library apache/arrow#3927
Merge ARROW-4983: [Plasma] Unmap memory upon destruction of the PlasmaClient apache/arrow#4001
Rebuild base docker image rayproject/arrow_linux_x86_64_base for latest arrow
Build new python wheels via https://github.com/ray-project/arrow-build (i.e. merge Update Arrow to https://github.com/apache/arrow/pull/4001 arrow-build#3)

AmplabJenkins · 2019-03-15T22:08:08Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/98/
Test PASSed.

guoyuhong · 2019-03-16T00:14:24Z

It looks like there is test failures in Mac. I cannot repro that error in my local machine. My colleague reported that some of plasma fails to start when the tests run some time in his old Mac system. I'm not sure whether this is the same problem.

AmplabJenkins · 2019-03-16T01:01:07Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12907/
Test FAILed.

AmplabJenkins · 2019-03-18T21:38:04Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/155/
Test PASSed.

AmplabJenkins · 2019-03-18T23:56:04Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/156/
Test PASSed.

AmplabJenkins · 2019-03-19T00:22:18Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12992/
Test PASSed.

pcmoritz · 2019-03-19T02:23:54Z

So the new wheels are now building successfully.

I also can't reproduce the macOS failure unfortunately. Increasing the timeout for connecting didn't help.

AmplabJenkins · 2019-03-19T02:26:13Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/162/
Test PASSed.

AmplabJenkins · 2019-03-19T02:57:16Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12996/
Test FAILed.

AmplabJenkins · 2019-03-19T05:22:40Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/13004/
Test PASSed.

AmplabJenkins · 2019-03-19T06:01:33Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/178/
Test PASSed.

guoyuhong · 2019-03-19T06:20:16Z

@pcmoritz You may also noticed that there are test failures in MacOS. It looks like that the plasma store fails to start when the travis test runs some time. This only happens in MacOS 10.11. @raulchen is investigating the same problem.

raulchen · 2019-03-19T06:36:05Z

I tried using the default osx version (xcode 9.4 and macOS 10.13), but CI still failed.

Plasma store log has the following error:

==> /tmp/ray/session_2019-03-19_06-06-41_89976/logs/plasma_store.err <==

external/plasma/cpp/src/plasma/store.cc:1093: Allowing the Plasma store to use up to 1.28849GB of memory.

external/plasma/cpp/src/plasma/store.cc:1120: Starting object store with directory /tmp and huge page support disabled

external/plasma/cpp/src/plasma/dlmalloc.cc:90: failed to ftruncate file /tmp/plasmayAfIcY

AmplabJenkins · 2019-03-19T06:50:07Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/180/
Test PASSed.

raulchen · 2019-03-19T08:04:06Z

Printed the error message: external/plasma/cpp/src/plasma/dlmalloc.cc:107: failed to ftruncate file /tmp/plasmawsYIAN No space left on device
@pcmoritz it seems that new arrow changes introduced memory leak on macOS?

Also, most of the tests use default plasma store size. This needs to be fixed.

AmplabJenkins · 2019-03-19T08:06:56Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/181/
Test PASSed.

AmplabJenkins · 2019-03-19T08:25:20Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/182/
Test PASSed.

raulchen · 2019-03-19T09:52:11Z

Setting default plasma memory to 100mb can fix this issue.

AmplabJenkins · 2019-03-19T10:27:01Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/183/
Test PASSed.

raulchen · 2019-03-21T06:19:06Z

Is this ready for merge now?

pcmoritz · 2019-03-21T07:53:39Z

We need to merge apache/arrow#4001, then rebuild the wheels from that, and then it can be merged. I expect we can merge this tomorrow.

AmplabJenkins · 2019-03-22T00:00:19Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/242/
Test PASSed.

AmplabJenkins · 2019-03-22T02:49:30Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/13172/
Test FAILed.

AmplabJenkins · 2019-03-22T20:18:05Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/253/
Test PASSed.

pcmoritz · 2019-03-22T22:10:26Z

Jenkins retest this please

AmplabJenkins · 2019-03-22T22:17:55Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/257/
Test PASSed.

AmplabJenkins · 2019-03-22T22:57:56Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/13204/
Test PASSed.

AmplabJenkins · 2019-03-22T23:33:05Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/258/
Test PASSed.

AmplabJenkins · 2019-03-23T01:16:50Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/13210/
Test FAILed.

AmplabJenkins · 2019-03-23T04:12:44Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/13213/
Test FAILed.

guoyuhong

Thanks for all the efforts!

AmplabJenkins · 2019-03-23T17:33:42Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/259/
Test PASSed.

AmplabJenkins · 2019-03-23T20:18:09Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/13216/
Test PASSed.

AmplabJenkins · 2019-03-27T22:51:52Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/290/
Test PASSed.

AmplabJenkins · 2019-03-28T01:12:47Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/13307/
Test PASSed.

AmplabJenkins · 2019-03-28T01:22:57Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/293/
Test PASSed.

AmplabJenkins · 2019-03-28T04:07:28Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/13311/
Test FAILed.

robertnishihara · 2019-03-28T05:37:58Z

It'd be nice to have a good way to keep these in sync in the future, though I guess we're going to have to do it manually?

I just merged this, but it occurred to me, we should add a comment at each place where we specify the arrow commit telling people that both places need to be updated simultaneously.

pcmoritz added 4 commits March 14, 2019 21:36

temp commit

f146655

update

013505d

update

9d79add

latest changes

ca0de95

pcmoritz changed the title ~~[WIP] Synchronize arrow versions and put changes upstream~~ [WIP] Synchronize arrow version and put changes upstream Mar 15, 2019

pcmoritz mentioned this pull request Mar 15, 2019

[Java] Build Java code with Bazel #4284

Merged

Update WORKSPACE

acb9559

pcmoritz added 2 commits March 18, 2019 16:54

update

f40d245

Merge branch 'arrow-bazel' of github.com:pcmoritz/ray-1 into arrow-bazel

5c5183c

update

fe23689

pcmoritz force-pushed the arrow-bazel branch from 3045942 to fe23689 Compare March 19, 2019 10:09

guoyuhong mentioned this pull request Mar 21, 2019

ARROW-4983: [Plasma] Unmap memory upon destruction of the PlasmaClient apache/arrow#4001

Closed

update

880a04a

Merge branch 'master' into arrow-bazel

599116a

pcmoritz changed the title ~~[WIP] Synchronize arrow version and put changes upstream~~ Synchronize arrow version and put changes upstream Mar 22, 2019

fix

6e05dcd

Merge branch 'master' into arrow-bazel

50d8f70

guoyuhong approved these changes Mar 23, 2019

View reviewed changes

Merge branch 'master' into arrow-bazel

74798eb

Update WORKSPACE

3d80115

robertnishihara merged commit 1bcb0b9 into ray-project:master Mar 28, 2019

robertnishihara deleted the arrow-bazel branch March 28, 2019 05:37

ashione pushed a commit to antgroup/ant-ray that referenced this pull request Apr 19, 2019

Synchronize arrow version and put changes upstream (ray-project#4385)

c6f9278

Synchronize arrow version and put changes upstream #4385

Synchronize arrow version and put changes upstream #4385

Uh oh!

Conversation

pcmoritz commented Mar 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AmplabJenkins commented Mar 15, 2019

Uh oh!

guoyuhong commented Mar 16, 2019

Uh oh!

AmplabJenkins commented Mar 16, 2019

Uh oh!

AmplabJenkins commented Mar 18, 2019

Uh oh!

AmplabJenkins commented Mar 18, 2019

Uh oh!

AmplabJenkins commented Mar 19, 2019

Uh oh!

pcmoritz commented Mar 19, 2019

Uh oh!

AmplabJenkins commented Mar 19, 2019

Uh oh!

AmplabJenkins commented Mar 19, 2019

Uh oh!

AmplabJenkins commented Mar 19, 2019

Uh oh!

AmplabJenkins commented Mar 19, 2019

Uh oh!

guoyuhong commented Mar 19, 2019

Uh oh!

raulchen commented Mar 19, 2019

Uh oh!

AmplabJenkins commented Mar 19, 2019

Uh oh!

raulchen commented Mar 19, 2019

Uh oh!

AmplabJenkins commented Mar 19, 2019

Uh oh!

AmplabJenkins commented Mar 19, 2019

Uh oh!

raulchen commented Mar 19, 2019

Uh oh!

AmplabJenkins commented Mar 19, 2019

Uh oh!

raulchen commented Mar 21, 2019

Uh oh!

pcmoritz commented Mar 21, 2019

Uh oh!

AmplabJenkins commented Mar 22, 2019

Uh oh!

AmplabJenkins commented Mar 22, 2019

Uh oh!

AmplabJenkins commented Mar 22, 2019

Uh oh!

pcmoritz commented Mar 22, 2019

Uh oh!

AmplabJenkins commented Mar 22, 2019

Uh oh!

AmplabJenkins commented Mar 22, 2019

Uh oh!

AmplabJenkins commented Mar 22, 2019

Uh oh!

AmplabJenkins commented Mar 23, 2019

Uh oh!

AmplabJenkins commented Mar 23, 2019

Uh oh!

guoyuhong left a comment

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Mar 23, 2019

Uh oh!

AmplabJenkins commented Mar 23, 2019

Uh oh!

AmplabJenkins commented Mar 27, 2019

Uh oh!

AmplabJenkins commented Mar 28, 2019

Uh oh!

AmplabJenkins commented Mar 28, 2019

Uh oh!

pcmoritz commented Mar 15, 2019 •

edited

Loading