Pandas DF column names can be complex objects in Python

[DavisVaughan]

Oddly, column names in a pandas DF (and lower level, in a dict()), can be complex objects, not just strings. This causes some issue with object conversion, but I think there is a solution with x$columns$format().

library(reticulate)

pd <- import("pandas", convert = FALSE)

d <- dict("col1" = list(1,2))

# Strange key where the key is actually a tuple
odd_key <- tuple("col1", "col2")
d[odd_key] = list(4,5)

d
#> {('col1', 'col2'): [4.0, 5.0], 'col1': [1.0, 2.0]}

# Strange df where the column name is actually a tuple
x = pd$DataFrame(data=d)

x
#>    col1  (col1, col2)
#> 0   1.0           4.0
#> 1   2.0           5.0

class(x$columns$values[[1]])
#> [1] "python.builtin.tuple"  "python.builtin.object"

py_to_r(x)
#> Error in py_call_impl(callable, dots$args, dots$keywords): KeyError: "['col2'] not in index"
#> 
#> Detailed traceback: 
#>   File "/Users/davisvaughan/.virtualenvs/r-reticulate/lib/python2.7/site-packages/pandas/core/frame.py", line 1991, in __getitem__
#>     return self._getitem_array(key)
#>   File "/Users/davisvaughan/.virtualenvs/r-reticulate/lib/python2.7/site-packages/pandas/core/frame.py", line 2035, in _getitem_array
#>     indexer = self.ix._convert_to_indexer(key, axis=1)
#>   File "/Users/davisvaughan/.virtualenvs/r-reticulate/lib/python2.7/site-packages/pandas/core/indexing.py", line 1214, in _convert_to_indexer
#>     raise KeyError('%s not in index' % objarr[mask])

# # debug with...
#debugonce(reticulate:::py_to_r.pandas.core.frame.DataFrame)
#py_to_r(x)

# Potential solution?
x$columns$format()
#> [u'col1', u'(col1, col2)']

Created on 2018-05-21 by the reprex package (v0.2.0).

[DavisVaughan]

I think a potential solution in py_to_r.pandas.core.frame.DataFrame might be to create a column_names object with py_to_r(x$columns$format()). Then you'd change this line to be:

names(converted) <- column_names

and this line to:

df <- converted[column_names]

My initial solution was to just change columns to what column_names is, but that doesn't quite work because when you extract and convert the columns of x here py_get_item() has to use the actual object in the column name to find that column, and not the string version of the column name.

[kevinushey]

One thing we might want to preserve is the ability to roundtrip the column names when sending the R data.frame back to Python after this conversion of names. Any thoughts on whether that's feasible, or whether it's worth exploring?

Do we need to emit a warning or anything similar if we end up translating a non-string column name in this way? (Just want to make sure the user is aware of any potential information loss)

[dfalbel]

Fixed in #277, although we haven't done any work to warn that this can break roundtrip casts. I'm going to close the issue as non-string column names is mostly an anti-pattern in pandas and not supported all around the Python community.

See eg:

dask/fastparquet#41
pandas-dev/pandas#10098