Skip to content

Commit 24abb0f

Browse files
changgyoopark-dbcloud-fan
authored andcommitted
[SPARK-50870][SQL] Add the timezone when casting to timestamp in V2ScanRelationPushDown
### What changes were proposed in this pull request? Add the timezone information to a cast expression when the destination type requires it. ### Why are the changes needed? When current_timestamp() is materialized as a string, the timezone information is gone (e.g., 2024-12-27 10:26:27.684158) which prohibits further optimization rules from being applied to the affected data source. For example, ``` Project [1735900357973433#10 AS current_timestamp()#6] +- 'Project [cast(2025-01-03 10:32:37.973433#11 as timestamp) AS 1735900357973433#10] +- RelationV2[2025-01-03 10:32:37.973433#11] xxx ``` -> This query fails to execute because the injected cast expression lacks the timezone information. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing tests. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#49549 from changgyoopark-db/SPARK-50870. Authored-by: changgyoopark-db <changgyoo.park@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
1 parent 1cdb918 commit 24abb0f

File tree

1 file changed

+6
-1
lines changed

1 file changed

+6
-1
lines changed

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -328,7 +328,12 @@ object V2ScanRelationPushDown extends Rule[LogicalPlan] with PredicateHelper {
328328
if (expression.dataType == expectedDataType) {
329329
expression
330330
} else {
331-
Cast(expression, expectedDataType)
331+
val cast = Cast(expression, expectedDataType)
332+
if (cast.timeZoneId.isEmpty && cast.needsTimeZone) {
333+
cast.withTimeZone(conf.sessionLocalTimeZone)
334+
} else {
335+
cast
336+
}
332337
}
333338

334339
def buildScanWithPushedAggregate(plan: LogicalPlan): LogicalPlan = plan.transform {

0 commit comments

Comments
 (0)