Skip to content

[WIP] fix: support format in from_unixtime #2769

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

[WIP] fix: support format in from_unixtime #2769

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/source/user-guide/latest/expressions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,12 +95,12 @@ incompatible expressions.
## Date/Time Functions

| Expression | SQL | Spark-Compatible? | Compatibility Notes |
| -------------- | ---------------------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------- |
| -------------- | ---------------------------- | ----------------- |----------------------------------------------------------------------------------------------------------------------|
| DateAdd | `date_add` | Yes | |
| DateSub | `date_sub` | Yes | |
| DatePart | `date_part(field, source)` | Yes | Supported values of `field`: `year`/`month`/`week`/`day`/`dayofweek`/`dayofweek_iso`/`doy`/`quarter`/`hour`/`minute` |
| Extract | `extract(field FROM source)` | Yes | Supported values of `field`: `year`/`month`/`week`/`day`/`dayofweek`/`dayofweek_iso`/`doy`/`quarter`/`hour`/`minute` |
| FromUnixTime | `from_unixtime` | No | Does not support format, supports only -8334601211038 <= sec <= 8210266876799 |
| FromUnixTime | `from_unixtime` | No | Supports only -8334601211038 <= sec <= 8210266876799 |
| Hour | `hour` | Yes | |
| Minute | `minute` | Yes | |
| Second | `second` | Yes | |
Expand Down
12 changes: 2 additions & 10 deletions spark/src/main/scala/org/apache/comet/serde/unixtime.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@
package org.apache.comet.serde

import org.apache.spark.sql.catalyst.expressions.{Attribute, FromUnixTime, Literal}
import org.apache.spark.sql.catalyst.util.TimestampFormatter

import org.apache.comet.CometSparkSessionExtensions.withInfo
import org.apache.comet.serde.QueryPlanSerde.{exprToProtoInternal, optExprWithInfo, scalarFunctionExprToProto}
Expand All @@ -36,17 +35,10 @@ object CometFromUnixTime extends CometExpressionSerde[FromUnixTime] {
inputs: Seq[Attribute],
binding: Boolean): Option[ExprOuterClass.Expr] = {
val secExpr = exprToProtoInternal(expr.sec, inputs, binding)
// TODO: DataFusion toChar does not support Spark datetime pattern format
// https://github.com/apache/datafusion/issues/16577
// https://github.com/apache/datafusion/issues/14536
// After fixing these issues, use provided `format` instead of the manual replacement below
val formatExpr = exprToProtoInternal(Literal("%Y-%m-%d %H:%M:%S"), inputs, binding)
val formatExpr = exprToProtoInternal(expr.format, inputs, binding)
val timeZone = exprToProtoInternal(Literal(expr.timeZoneId.orNull), inputs, binding)

if (expr.format != Literal(TimestampFormatter.defaultPattern)) {
withInfo(expr, "Datetime pattern format is unsupported")
None
} else if (secExpr.isDefined && formatExpr.isDefined) {
if (secExpr.isDefined && formatExpr.isDefined) {
val timestampExpr =
scalarFunctionExprToProto("from_unixtime", Seq(secExpr, timeZone): _*)
val optExpr = scalarFunctionExprToProto("to_char", Seq(timestampExpr, formatExpr): _*)
Expand Down
12 changes: 4 additions & 8 deletions spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1769,17 +1769,13 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
val where = "where _5 BETWEEN -8334601211038 AND 8210266876799"
checkSparkAnswerAndOperator(s"SELECT from_unixtime(_5) FROM $table $where")
checkSparkAnswerAndOperator(s"SELECT from_unixtime(_8) FROM $table $where")
// TODO: DataFusion toChar does not support Spark datetime pattern format
// https://github.com/apache/datafusion/issues/16577
// https://github.com/apache/datafusion/issues/14536
// After fixing these issues, change checkSparkAnswer to checkSparkAnswerAndOperator
checkSparkAnswer(s"SELECT from_unixtime(_5, 'yyyy') FROM $table $where")
checkSparkAnswer(s"SELECT from_unixtime(_8, 'yyyy') FROM $table $where")
checkSparkAnswerAndOperator(s"SELECT from_unixtime(_5, 'yyyy') FROM $table $where")
checkSparkAnswerAndOperator(s"SELECT from_unixtime(_8, 'yyyy') FROM $table $where")
withSQLConf(SESSION_LOCAL_TIMEZONE.key -> "Asia/Kathmandu") {
checkSparkAnswerAndOperator(s"SELECT from_unixtime(_5) FROM $table $where")
checkSparkAnswerAndOperator(s"SELECT from_unixtime(_8) FROM $table $where")
checkSparkAnswer(s"SELECT from_unixtime(_5, 'yyyy') FROM $table $where")
checkSparkAnswer(s"SELECT from_unixtime(_8, 'yyyy') FROM $table $where")
checkSparkAnswerAndOperator(s"SELECT from_unixtime(_5, 'yyyy') FROM $table $where")
checkSparkAnswerAndOperator(s"SELECT from_unixtime(_8, 'yyyy') FROM $table $where")
}
}
}
Expand Down
Loading