Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DataFrame] Add Parquet Support in Build Process #1531

Merged
merged 8 commits into from
Feb 16, 2018
Merged

[DataFrame] Add Parquet Support in Build Process #1531

merged 8 commits into from
Feb 16, 2018

Conversation

simon-mo
Copy link
Contributor

What do these changes do?

This PR adds the following to ray master

  1. Download parquet-cpp
  2. Build it inside a conda environement.
  3. Let pyarrow know that it needs to be built with parquet.

Related issue number

#1518

Note

This PR is likely to fail in Travis/Jenkins due to environment/dependency issue. There will be more fixes once I can view the travis result.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3653/
Test FAILed.

@simon-mo
Copy link
Contributor Author

  • It seems the cause is -- libevent NOT found.. Let me try to download and compile libevent before parquet built process.

@robertnishihara robertnishihara mentioned this pull request Feb 11, 2018
Closed
@pcmoritz
Copy link
Contributor

I think the macOS build is failing because bison is not the right version. It needs to be brew installed, see apache/parquet-cpp@9f82afe

Let me know if you need help with that!

@simon-mo
Copy link
Contributor Author

simon-mo commented Feb 13, 2018 via email •
edited
Loading

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3687/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3689/
Test FAILed.

@simon-mo
Copy link
Contributor Author

Hmm... seems to be not unrelated ray tune issue:

ray.tune.error.TuneError: Unknown trainable: PG

@pcmoritz
Copy link
Contributor

Hey @simon-mo: That failure is not a problem, but the one on travis is (it's not finding boost).

I think you need to set the boost location with something like this:

TP_DIR=$(cd "$(dirname "${BASH_SOURCE:-$0}")"; pwd)

BOOST_ROOT=$TP_DIR/boost \
ARROW_HOME=$TP_DIR/arrow/cpp \
cmake ...

(see also https://github.com/ray-project/ray/blob/master/build.sh)

Let me know how it goes!

@robertnishihara
Copy link
Collaborator

Note that there is currently a difference in how we do things on Mac and Linux (though we should probably just get rid of this difference).

On Linux we're compiling Boost. On Mac we're doing brew install boost.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3716/
Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3719/
Test PASSed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3724/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3725/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3726/
Test FAILed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3729/
Test PASSed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3741/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3744/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3748/
Test FAILed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3747/
Test PASSed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3751/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3754/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3756/
Test FAILed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3757/
Test PASSed.

@pcmoritz
Copy link
Contributor

Tests pass in https://amplab.cs.berkeley.edu/jenkins/job/Ray-PRB/3757/ and https://travis-ci.com/robertnishihara/ray-private-travis/builds/66060598

@pcmoritz pcmoritz self-requested a review February 16, 2018 02:02
@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3758/
Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/3759/
Test PASSed.

@pcmoritz pcmoritz changed the title [DataFrame] [WIP] Add Parquet Support in Build Process [DataFrame] Add Parquet Support in Build Process Feb 16, 2018
@pcmoritz pcmoritz merged commit a24cc28 into ray-project:master Feb 16, 2018
robertnishihara
robertnishihara reviewed Feb 16, 2018
View reviewed changes
src/thirdparty/build_parquet.sh
TP_DIR=$(cd "$(dirname "${BASH_SOURCE:-$0}")"; pwd)

if [ "$unamestr" == "Darwin" ]; then
brew update > /dev/null
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we shouldn't be doing this here

robertnishihara
robertnishihara reviewed Feb 16, 2018
View reviewed changes
docker/base-deps/Dockerfile
@@ -4,7 +4,7 @@
FROM ubuntu:xenial
RUN apt-get update \
&& apt-get install -y vim git wget \
&& apt-get install -y cmake pkg-config build-essential autoconf curl libtool unzip
&& apt-get install -y cmake pkg-config build-essential autoconf curl libtool unzip flex bison
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we'll also need to update the installation instructions

robertnishihara
robertnishihara reviewed Feb 16, 2018
View reviewed changes
src/thirdparty/build_parquet.sh
brew install bison
export OPENSSL_ROOT_DIR=/usr/local/opt/openssl
export LD_LIBRARY_PATH=/usr/local/opt/openssl/lib:$LD_LIBRARY_PATH
export PATH="/usr/local/opt/bison/bin:$PATH"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These three lines don't feel very robust..

This was referenced Feb 16, 2018
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@robertnishihara robertnishihara robertnishihara left review comments

@pcmoritz pcmoritz pcmoritz approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@simon-mo @AmplabJenkins @pcmoritz @robertnishihara