FEAT: Allow custom engine options in to_sql using ModinDatabaseConnection

[bensolom]

Hi
I was wondered if is it possible to init the pyODBC engine with fast_executemany flag and pass it to modin.to_sql function.
It has great performance in pandas, ex:

engine_obj = create_engine(engine_str, fast_executemany=True)
pandas_df.to_sql(name="table", con=engine_obj)

Thank you
Ben

[mvashishtha]

@bensolom thank you for the suggestion; this need has come up in different forms before. Currently, we require that users pass in a connection string to to_sql because we can't pickle a connection object (see #4048). However, users often need to make each connection with some extra parameters, e.g. fast_executemany here (as it happens, I have seen the need for this exact parameter before).

the solution is to copy our solution for read_sql: users can pass in a pickleable ModinDatabaseConnection object that describes how to make each connection for the write. I actually implemented this feature in commits on top of modin 0.8.3, (in particular see this one) but we have yet to merge this feature into Modin.

Unfortunately I can't get to this right now. Contributions are welcome. cc @modin-project/modin-contributors @modin-project/modin-core

[bensolom]

@bensolom thank you for the suggestion; this need has come up in different forms before. Currently, we require that users pass in a connection string to to_sql because we can't pickle a connection object (see #4048). However, users often need to make each connection with some extra parameters, e.g. fast_executemany here (as it happens, I have seen the need for this exact parameter before).

the solution is to copy our solution for read_sql: users can pass in a pickleable ModinDatabaseConnection object that describes how to make each connection for the write. I actually implemented this feature in commits on top of modin 0.8.3, (in particular see this one) but we have yet to merge this feature into Modin.

Unfortunately I can't get to this right now. Contributions are welcome. cc @modin-project/modin-contributors @modin-project/modin-core

Thank you @mvashishtha