Implement a maximum pandas compatibility mode

[gshimansky]

Some recent PRs introduced known difference in behavior between Modin and Pandas. I know of at least two:

1. Different order of columns in groupby.__getitem__ FIX-#3252: Fix GroupBy.__getitem__ #3298 (comment)
2. Index instead of MultiIndex in DataFrame.value_counts FIX-#1294: fixed 'value_counts' implementation #2730 (comment)

Workarounds for differences in both cases above are trivial but impair performance. I suggest to add a setting called something like MODIN_MAXIMUM_PANDAS_COMPAT that would implement workarounds at the expense of performance and make functions behavior in the closest possible way to how Pandas implements them. In case some user's application breaks when he replaces import pandas with import modin.pandas such setting would allow to quickly fix problems that happen because of incompatibility.

[amyskov]

I think such mode can be very helpful for #3288. In the case if heterogeneous data is imported, partitions data can have different type and casting it to proper data type can be expensive. In the case if such discrepancy is unimportant, this casting can be skipped.

[dchigarev]

Another couple of cases that deviate from pandas:

1. PandasQueryCompiler.melt - pandas melt function preserves the order of rows of the id_vars column, in the current implementation order of rows is chaotic. Could be resolved by reindexing the result. Original explanation of this difference.
   

   modin/modin/backends/pandas/query_compiler.py

   Lines 2027 to 2029 in 0a64275

   	ErrorMessage.missmatch_with_pandas(
   	operation="melt", message="Order of rows could be different from pandas"
   	)

2. PandasQueryCompiler.pivot_table - the results column labels have to be sorted in ascending order, currently unsorted. Could be resolved by sorting the column axis. Original report of this diff.
   

   modin/modin/backends/pandas/query_compiler.py

   Lines 2936 to 2939 in 0a64275

   	ErrorMessage.missmatch_with_pandas(
   	operation="pivot_table",
   	message="Order of columns could be different from pandas",
   	)

[dchigarev]

I think we should start from determining some kind of criterios, that will describe expected differences between strict and non-strict modes. With their help, we can easily modify our current comparators in tests. I can suggest that the following is not guaranteed in the non-strict mode:

1. Order of rows and columns could be different from pandas. Comparator changes: both axes have to be sorted before comparison with pandas result.
2. Object type of axis labels could be different, however, its content is guarantee to be the same (for except of the order). Comparator changes: before comparison, cast index object type to align with pandas type if required.
3. Probably some of the default function parameters could be different to ensure the best performance? For example, if the handling of the default dropna=True parameter is complicated it could be set to dropna=False.
4. Data type of columns could be different. This actually needs to be clarified, we need to agree on which data-types we allow deviations and what kind this deviation could be. I suggest that the initial alloweness table is:

modin type	pandas type	is allowed
any numeric	any numeric	+
any numeric	object	+
object	any numeric	-
any datetime	any datetime	+
any datetime	object	+
object	any datetime	-
any category	any category	+
any category	object	+
object	any category	?

cc @modin-project/modin-core @modin-project/modin-omnisci

[RehanSD]

Hi @dchigarev @gshimansky @amyskov has this issue been resolved?

[Garra1980]

No, this is something we discussed some time ago but never implemented. It is a performance-related feature and we may return to this idea in the future