concat on columns fails with multi-index when index names are not overlapping

[eavidan]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS 10.15.7
• Modin version (modin.__version__): 0.8.2
• Python version: 3.7
• Code we can use to reproduce: yes

Describe the problem

Using concat columns on 2 data frames with multi-index and no overlapping index names fails with ValueError: cannot join with no overlapping index names. Same operation works using pandas

Source code / logs

The following works with pandas but fails with modin

df1 = pd.DataFrame({('s1', 426, 6, -3): 210}, index=[["a"], ["b"], ["c"], ["d"]])
df2 = pd.DataFrame({('s2', 194, -3, -5): None}, index=[["x"], ["b"], ["c"], ["d"]])
df2.index.names = ['x', 'y', 'z', 'w']
merged = pd.concat([df1, df2], axis=1)

[YarShev]

Hi @eavidan , thanks for posting! @dchigarev , can you look at this? The traceback seems to be similar to that what are you trying to resolve in #2443 .

[dchigarev]

@YarShev
This issue is different from #2443 and #2378, although it has the same root cause. The problem is in our modin_frame._concat function and the way we're using it.

Modins frame 'concat' is not a simple 'concat' like in pandas, it combines functionality of inserting, concating, joining, merging, and broadcasting for binary operations. And the problem is, that it tries to apply the logic of "joining" that will fit for all that cases, which seems to be not very good approach (going by all issues related with _concat and _copartition that was recently found).

Merge, join, concat, insert, binary_ops — they all have different logic of "joining" axes in pandas, modin tries to use common approach.

#2443 and #2378 tries to solve that issue for insert and binary_ops. It seems that concat will be the next in that row.

P.S.
How our modin_frame._concat differs from pandas.concat:
Pandas depending on joining type (inner or outer) does either Index.intersection or Index.union. Call-stack would be:
concat -> _Concatenator._get_new_axes -> _Concatenator._get_comb_axis -> Indexes.api.get_objs_combined_axis -> Indexes.api._get_combined_index

Modin depending on joining type does Index.join(how="inner") or Index.join(how="outer"), which as we can see works differently from pandas in reported case.

[pyrito]

I am able to reproduce this bug on the latest master.