Incorrect string cast for time and interval types

[linhr]

The string representation for time and interval types does not match the Spark behavior in cast(a AS string).

These test cases are related:

• test_create_data_frame_to_pandas_day_time_internal
• test_create_data_frame_to_pandas_timestamp_ntz

[SemyonSinchenko]

I'm just interesting, why not to use apache datafusion comet crate? It already contains implementations for most spark-compatible expressions & operators. For example, this part should cover that issue: https://github.com/apache/datafusion-comet/blob/main/native/spark-expr/src/conversion_funcs/cast.rs

[linhr]

I'm just interesting, why not to use apache datafusion comet crate? It already contains implementations for most spark-compatible expressions & operators. For example, this part should cover that issue: https://github.com/apache/datafusion-comet/blob/main/native/spark-expr/src/conversion_funcs/cast.rs

Glad you brought up this! It's possible to depend on Comet directly but there would be some maintenance work due to different release schedule of Comet and DataFusion. (The DataFusion version in Comet and Sail must be the same.)

We have been part of the ongoing conversation to move Spark functions and expressions from Comet to the core DataFusion repo. Once the crate is set up, we can depend on it to support missing features in Sail. The hope is that Sail and other interested parties would also contribute to the shared Spark functions crate in DataFusion.

Here are the related discussions in DataFusion:

• [DISCUSSION] Add separate crate to cover spark builtin functions apache/datafusion#5600
• feat: Add datafusion-spark crate apache/datafusion#14392

[linhr]

This is fixed in #448.