Pyarrow is not properly detected after importing ray

[devin-petersohn]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): WSL
• Ray installed from (source or binary): pip
• Ray version: 0.6.1
• Python version: 3.6
• Exact command to reproduce: pandas.DataFrame({"col1":[1,2,3], "col2":[2,4,5]}).to_parquet("test.parquet")

Describe the problem

The string version of the pyarrow shipped with ray is not able to be detected by pandas.

Source code / logs

String version in 0.6.1 is:

In [1]: import ray
im
In [2]: import pyarrow as pa

In [3]: pa.__version__
Out[3]: '0.1.dev3212+gc347cd5'

When trying to use pandas to write a parquet file, it does not detect that a valid pyarrow is installed because it is looking for pyarrow>=0.4. Here is the pandas command that reproduces it:

import ray
import pandas
pandas.DataFrame({"col1":[1,2,3], "col2":[2,4,5]}).to_parquet("test.parquet")

[robertnishihara]

Thanks for reporting this, I can reproduce it. The error I get when running those lines is

In [3]: pandas.DataFrame({"col1":[1,2,3], "col2":[2,4,5]}).to_parquet("test.parquet")
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-3-570c900b574f> in <module>()
----> 1 pandas.DataFrame({"col1":[1,2,3], "col2":[2,4,5]}).to_parquet("test.parquet")

~/anaconda3/lib/python3.6/site-packages/pandas/core/frame.py in to_parquet(self, fname, engine, compression, **kwargs)
   1647         from pandas.io.parquet import to_parquet
   1648         to_parquet(self, fname, engine,
-> 1649                    compression=compression, **kwargs)
   1650 
   1651     @Substitution(header='Write out the column names. If a list of strings '

~/anaconda3/lib/python3.6/site-packages/pandas/io/parquet.py in to_parquet(df, path, engine, compression, **kwargs)
    224         Additional keyword arguments passed to the engine
    225     """
--> 226     impl = get_engine(engine)
    227     return impl.write(df, path, compression=compression, **kwargs)
    228 

~/anaconda3/lib/python3.6/site-packages/pandas/io/parquet.py in get_engine(engine)
     27             pass
     28 
---> 29         raise ImportError("Unable to find a usable engine; "
     30                           "tried using: 'pyarrow', 'fastparquet'.\n"
     31                           "pyarrow or fastparquet is required for parquet "

ImportError: Unable to find a usable engine; tried using: 'pyarrow', 'fastparquet'.
pyarrow or fastparquet is required for parquet support

And as you did in the above commit, manually setting pa.__version__ = '0.11.0' fixes the problem.

I guess pandas isn't compatible with versions of pyarrow that are built from source?

[devin-petersohn]

It is a recent issue, because pip install ray==0.6 results is pa.__version__ of '0.11.1.dev110+gb4f7ed6d'.

Pandas does not detect any pyarrow<0.4 because of compatibility issues.

[pcmoritz]

@robertnishihara I've been looking into this, currently, the version number of our pyarrow is malformed, it thinks the version is 0.1 and that's where the trouble comes from.

[pcmoritz]

Another thing I found out while working on this: While setting the version correctly fixes the problem for the wheels, a custom built ray will fail with

In [1]: import ray                                                                                                                                                                   

In [2]: import pyarrow.parquet                                                                                                                                                       
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-9281362b9246> in <module>
----> 1 import pyarrow.parquet

~/ray/python/ray/pyarrow_files/pyarrow/parquet.py in <module>
     28 import pyarrow as pa
     29 import pyarrow.lib as lib
---> 30 import pyarrow._parquet as _parquet
     31 
     32 from pyarrow._parquet import (ParquetReader, RowGroupStatistics,  # noqa

ImportError: /home/ubuntu/ray/python/ray/pyarrow_files/pyarrow/libparquet.so.12: undefined symbol: _ZNK5boost16re_detail_10680031cpp_regex_traits_implementationIcE17transform_primaryEPKcS4_

[pcmoritz]

fixed by #3760