Migrate existing UDFs to PPLFuncImpTable

[yuancu]

Description

Refactor existing math, datetime, text, condition UDFs to PPLFuncImpTable.

Why self-implementing instead of leveraging Calcite builtin functions

• Not implemented by calcite

  • CONV
  • CRC32 (implemented in a newer version of calcite)
  • E
  • LOCATE

• Different definition

  • MOD & DIVIDE: return types (ours return wider type; calcite implementation return divisor type); handling of divisor 0 (ours return null)

• Datetime related: we use UDT and accepts STRING most of the time.

Note:

• UDAFs are kept untouched.

TODOs:

• Fix filled date problems related to function properties.

Related Issues

Resolves #3524

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[qianheng-aws]

It's confusing to use SqlTypeName.TIME here, temporalType here is actually UDT whose real SqlTypeName is VARCHAR actually. As discussed, we can determine the return type based on whether temporalType is AbstractExprRelDataType with getUDT equal to EXPR_TIME

Could we remove this API convertRelDataTypeToSqlTypeName? Its logic doesn't make sense.

[yuancu]

I removed the convertRelDataTypeToSqlTypeName API and adopted convertRelDataTypeToExprType to compare types, so that I can use a more unified interface to compare not only UDT but also builtin types like String. The converted ExprType is also widely used in the conversion to ExprValue (to reuse v2's implementation). Is the new implementation more logically acceptable?

[qianheng-aws]

Seems never used.

[yuancu]

Removed. It was only used in previous implementations.

[penghuo]

calcite has sqrt function?

0: jdbc:calcite:model=src/test/resources/mode> select sqrt(9);
+--------+
| EXPR$0 |
+--------+
| 3.0    |
+--------+
1 row selected (0.588 seconds)

[yuancu]

SQRT is only declared in SqlStdOperatorTable, but not implemented in RexImpTable. I.e. there isn't a statement like defineMethod(SQRT, BuiltInMethod.SQRT.method, NullPolicy.STRICT) in RexImpTable.

If I register sqrt to the calcite SQRT operator, it will throw an error at runtime:

...
java.lang.RuntimeException: java.sql.SQLException: Error while preparing plan [LogicalProject(name=[$0], age=[$5], month=[$3])
              LogicalFilter(condition=[AND(=(SQRT(POWER($5, 2)), 30.0E0), =(CBRT(POWER($3, 3)), 4))])
    ...
    java.lang.IllegalStateException: Unable to implement EnumerableCalc(expr#0..11=[{inputs}], expr#12=[2], expr#13=[POWER($t5, $t12)], expr#14=[SQRT($t13)], expr#15=[30.0E0:DOUBLE], expr#16=[=($t14, $t15)], expr#17=[3], expr#18=[POWER($t3, $t17)], expr#19=[CBRT($t18)], expr#20=[4], expr#21=[=($t19, $t20)], expr#22=[AND($t16, $t21)], name=[$t0], age=[$t5], month=[$t3], $condition=[$t22]): rowcount = 2.25, cumulative cost = {102.25 rows, 2801.0 cpu, 0.0 io}, id = 1388
     CalciteEnumerableIndexScan(table=[[OpenSearch, opensearch-sql_test_index_state_country]]): rowcount = 100.0, cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io}, id = 1353

[yuancu]

@qianheng-aws pointed out that SQRT is converted to POWER(*, 0.5). I've removed the sqrt udf and register the operator to power with 0.5 as the second parameter instead.

[LantaoJin]

@qianheng-aws pointed out that SQRT is converted to POWER(*, 0.5). I've removed the sqrt udf and register the operator to power with 0.5 as the second parameter instead.

Since Calcite already has the SQRT function definition in SqlStdOperatorTable and internal implementation (converting to POWER(*, 0.5) is also a kind of implementation), why do we still need to explicitly add this conversion in our side? @yuancu

[yuancu]

In our implementation, when using operators from Calcite, their implementations have to exist in RexImpTable. Otherwise, it will throw an error like cannot translate call SQRT. See RexToLixTranslater.java#L1428.

In Calcite SQL, the conversion happens in SqlToRelConverter. See SqlNodeToRexConverterImpl.java#L59. However, we do not have access to SqlCall in our implementation. Therefore, we cannot directly reuse Calcite's conversion of SQRT function.

[penghuo]

@yuancu We can similar convert in PPLFuncImplTable.

      register(SQRT,
          ((FunctionImp1)
          (builder, arg1) -> builder.makeCall(SqlStdOperatorTable.POWER, arg1,
              builder.makeApproxLiteral(BigDecimal.valueOf(0.5)))));

@qianheng-aws IMO, PPLFuncImplTable support 2 ways to define function. 1) Use existing SqlFunctions. 2) UDF. Is it correct?

[yuancu]

Yes. I have registered SQRT in this fashion.