Skip to content

Conversation

@xinyual
Copy link
Contributor

@xinyual xinyual commented Feb 27, 2025

Description

In this PR we add several UDFs and UDAFs. We add implement for functions which exist in both index PPL and Spark PPL from #3310.

  1. For those we can find corresponding function in calcite built-in functions, we map it and do some arguments modification if needed.
  2. For those we cannot find corresponding function in calcite built-in functions, we follow the design in [FEATURE] Calcite Engine Framework: General function (UDF/UDAF) framework #3310 to implement it.
  3. We add standalone UT.

Briefly, we implement all math/string/condition functions list in #3310 plus percentile and take aggregation.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

xinyual and others added 14 commits February 24, 2025 10:42
Signed-off-by: xinyual <xinyual@amazon.com>
Signed-off-by: xinyual <xinyual@amazon.com>
Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>
Signed-off-by: xinyual <xinyual@amazon.com>
Signed-off-by: Xinyuan Lu <xinyual@amazon.com>
Map opensearch math functions to calcite implementations
Signed-off-by: xinyual <xinyual@amazon.com>
Signed-off-by: xinyual <xinyual@amazon.com>
Signed-off-by: xinyual <xinyual@amazon.com>
Signed-off-by: xinyual <xinyual@amazon.com>
Signed-off-by: xinyual <xinyual@amazon.com>
Signed-off-by: xinyual <xinyual@amazon.com>
Signed-off-by: xinyual <xinyual@amazon.com>
Signed-off-by: xinyual <xinyual@amazon.com>
Signed-off-by: xinyual <xinyual@amazon.com>
@LantaoJin
Copy link
Member

LantaoJin commented Feb 27, 2025

  1. Rename the title to include what kind functions this PR added.
  2. Add CalcitePPLBuiltinFunctionTest extends CalcitePPLAbstractTest for UT (this is important to check the plan generated by Calcite, example in Implement ppl join command with Calcite #3364)
  3. Add more ITs refer the PPL query cases in https://github.com/opensearch-project/opensearch-spark/blob/main/integ-test/src/integration/scala/org/opensearch/flint/spark/ppl/FlintSparkPPLBuiltinFunctionITSuite.scala and other FlintSparkPPLXXFunctionITSuite

@LantaoJin LantaoJin added the calcite calcite migration releated label Feb 27, 2025
Signed-off-by: xinyual <xinyual@amazon.com>
Signed-off-by: xinyual <xinyual@amazon.com>
@xinyual xinyual changed the title Add partial udf Add partial udf:String/condition/math built-in functions Feb 28, 2025
@LantaoJin
Copy link
Member

@xinyual can we close this?

@xinyual
Copy link
Contributor Author

xinyual commented Mar 7, 2025

We already have separate PR for them. So I close this PR.

@xinyual xinyual closed this Mar 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

calcite calcite migration releated

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants