Typeerror: can only merge series or dataframe objects a class 'pandas.core.frame.dataframe' was passed

[quinnnnnnnnn]

File "D: \Python程序\因子入库0421\factor_ top_ pure_ czl_ 0421_ gtja191.py", line 323,
in BarraDataPreparelong

BarraMatrix long = pd . merge barraclass, industry，on=['code'，'date'], how= ' inner' )#在barra因子矩阵右侧追加行业哑变量矩阵

File "C: \Users \Administrator\AppData \Roaming \Python\Python38 \site - packages\modin\pandas \general.py", line 84,in merge
raise TypeError(
TypeError: Can only merge Series or DataFrame objects, a <class
pandas. core. frame .DataFrame' > was passed

anyone kowns how to deal with it? 0.0

[devin-petersohn]

@quinnnnnnnnn Are you mixing pandas and Modin together? It seems like that might be the cause.

[YarShev]

@devin-petersohn, do you think we should allow our merge implementation to take in a pandas DataFrame as we allow to concatenate pandas DataFrame(s)?

[devin-petersohn]

It is not recommended to mix Modin and pandas. Pandas will not accept Modin inputs, so I don't think we should allow this interaction.

Separately, we can create something that does make them compatible, but users should opt-in to this behavior IMO.

[YarShev]

I agree with both your statements. This should be put into the list for 1.0.

Aside: although passing a pandas DataFrame in a function that uses broadcast join algorithm would speed up the flow 😄

[vnlitvinov]

I think we may introduce a special descendant from modin.DataFrame which would be a very thin wrapper around pandas.DataFrame so Modin user would be able to construct this special object from their pandas df (serving as opt-in), and this special descendant wouldn't be distributing this dataframe, somewhat like PandasOnPython does.

[RehanSD]

Hi folks - do we plan to add support for intermixing pandas types? I think we've had this discussion before, and ultimately landed on not adding interoperability, but I'm curious to see if folks think we should add it!

[adrienpacifico]

I wanted to use Modin as a drop-in replacement for pandas.
However, there is a function in my data-preparation script that calls external functions that returns pandas DataFrame (let's say an API that returns a pandas DataFrame)

As a result, Modin is not a simple drop-in replacement since I have to refactor my code (i.e., modin.pandas.DataFrame(pandas_dataframe), and it cannot easily be automatized ). However, a solution that allows Modin to perform operations with pandas Series and DataFrame would likely allow those kinds of operations without having to think about it.

It is not recommended to mix Modin and pandas. Pandas will not accept Modin inputs, so I don't think we should allow this interaction.

@devin-petersohn Could you develop that? Modin dataframe will not be compatible with pandas dataframes, but I do not see the potential harm in having Modin convert pandas dataframes to modin dataframes.

A code example that I would like to be able to use without refactoring:

import modin.pandas as pd  # replace previous pandas import
from my_company_data_utils import get_dataframe_of_something

df_pandas = get_dataframe_of_something() # returns a pandas dataframe
df_modin = pd.DataFrame({"A":[1,2,3,4]}) # Creates a Modin dataframe

df_modin.merge(df_pandas, on=["A"])
# or 
pd.merge(df_pandas, df_modin, on = ["A"])

However, probably that having df_pandas.merge(df_modin, on=["A"]) would likely imply modifying the pandas library, or to do hacky stuff such that all pandas dataframes be converted into Modin dataframes automatically.

Separately, we can create something that does make them compatible, but users should opt-in to this behavior IMO.

That would be super nice!

[mvashishtha]

@adrienpacifico thank you for commenting here. There is some demand to have interoperability beyond calling modin_df._to_pandas() and/or modin.pandas.DataFrame(pandas_df). We still haven't decided how that will work, though. I'll move the priority of this issue up to P2 for now.