ARROW-1976: [Python] Handling unicode pandas columns on parquet.read_table

[Licht-T]

This closes ARROW-1976.

[xhochy]

This should also be six.text_type so that we get unicode in Python 2. Probably, using .decode('utf-8') might also be the better option.

[wesm]

There is also frombytes in pyarrow.compat

[Licht-T]

Thanks @xhochy, fixed!

[xhochy]

frombytes works only on bytes. Thus the above code is valid in Python 2 but breaks the unittests for Python 3. Removing the str should fix this.

[wesm]

I'm looking at this patch to get it passing

[Licht-T]

Sorry @wesm! Ths totally slipped my mind!

[wesm]

Can someone look at getting this ready to merge? I'm drowning trying to get on top of the PR queue and e-mail firehose. @cpcloud @xhochy

[ylogx]

I'd like to help. If it's fine with @Licht-T, I can pull his branch and create a new PR to add the required changes there.

[simnyatsanga]

@Licht-T @wesm @ylogx I made a PR with the requested feedback in this main PR here: Licht-T#1 . Hopefully this is ok.

[xhochy]

Together with the changes from @simnyatsanga this is good to go.

[wesm]

Cool, I just merged the changes, will await the CI to run

[wesm]

This is still failing

[simnyatsanga]

I'm looking at getting the failing tests to pass. Specifically one of the failing tests looks like this
https://github.com/apache/arrow/blob/master/python/pyarrow/tests/test_convert_pandas.py#L141

def test_multiindex_columns(self):
        columns = pd.MultiIndex.from_arrays([
            ['one', 'two'], ['X', 'Y']
        ])
        df = pd.DataFrame([(1, 'a'), (2, 'b'), (3, 'c')], columns=columns)
        _check_pandas_roundtrip(df, preserve_index=True)

The pandas roundtrip is failing in this branch on this line: https://github.com/apache/arrow/blob/master/python/pyarrow/pandas_compat.py#L166
Because the column_name (which is part of a MultiIndex) is coming in as a tuple instead of a string. On master the column_name is indeed coming in as a string. I'm trying to figure out, what changes on this branch would cause this.

[cpcloud]

@Licht-T @wesm @simnyatsanga I'm taking a look at this now.

[cpcloud]

This is failing because of an assumption about the behavior of frombytes. Pushing up a fix shortly.

[cpcloud]

@Licht-T closing in favor of #1553