SNOW-2216201: Allow for passing pyarrow schema to write_pandas

[Seminko]

What is the current behavior?

I have a dataframe where one column holds python dicts. When I want to import it to snowflake using write_pandas it fails on memory (basically eats up all the memory and then fails on malloc).

To solve this I have to define a custom pyarrow schema. However I am unable to pass it as a kwarg to write_pandas because schema is already a named parameter.

Here's the complete flow:

pandas_tools.py
def write_pandas
chunk.to_parquet(chunk_path, compression=compression, **kwargs) # <-- if I pass schema=my_pyarrow_schema here, it works fine, but I cant because write_pandas already has a parameter named schema

What is the desired behavior?

Be able to pass pyarrow schema to write_pandas

How would this improve snowflake-connector-python?

Allow for non-standard dfs to be imported into SF.

References and other background

No response

[sfc-gh-dszmolka]

hi @Seminko and thanks for submitting this enhancement proposal. The team will take a look and see if it's something to be considered to be implemented. For transparency; it might not happen very soon (focus is on bugs and critical feature implementations) so if it's blocking for your use-case, it's highly recommended to reach out to your Snowflake Account Team and let them know you need this feature. Alternatively if its within your possibilities, you can also submit a PR. Thank you for understaning.