depr(python): Deprecate overwrite_schema parameter for DataFrame.write_delta

[stinodego]

deltalake changed their public API. They no longer have an overwrite_schema parameter. We exposed this parameter directly in our API.

I removed this parameter to defer directly to the deltalake API through delta_write_options. Users can pass delta_write_options={"schema_mode": "overwrite"} to get the same result.

Also had to update some tests as deltalake handles timezones differently now.

@ion-elgreco FYI

[ion-elgreco]

Sorry forgetting to give some heads up, just did the release in the train:P Since this release was breaking for sure, I've added a feature to finally handled datetimes with timezones properly.

Maybe it could make sense to do it the same way as we did, add schema_mode as additional parameter in the function, in stead of passing it through delta_write_options. I expect schema_mode to be quite commonly used.

[stinodego]

Maybe it could make sense to do it the same way as we did, add schema_mode as additional parameter in the function, in stead of passing it through delta_write_options. I expect schema_mode to be quite commonly used.

I considered this, as it was the original reason for including the parameter directly in our API. However, since there are now multiple options (merge, overwrite, None), possibly more added in the future, and a tie-in to the underlying engine (merge doesn't work with the pyarrow-engine), this is now getting too much of a responsibility to put on our own API.

[ion-elgreco]

@stinodego alright! Shall we maybe bring it back once the pyarrow engine gets deprecated and the overwrite_schema parameter in write_deltalake?

These things are likely to happen in deltalake 1.0 release

[stinodego]

@stinodego alright! Shall we maybe bring it back once the pyarrow engine gets deprecated and the overwrite_schema parameter in write_deltalake?

These things are likely to happen in deltalake 1.0 release

Sounds good to me! We can revisit when the 1.0 release is out.

However, I am very wary of tying our public API to that of a dependency, exactly for the reason that caused this PR.