Skip to content

Conversation

@qianheng-aws
Copy link
Collaborator

Description

This PR includes 2 changes:

  1. Fix PredicateAnalyzer for in and notIn, it build the DSL query with redundant brackets because of calling wrong API.
  2. For the tables used in this IT, change some fields from text type to keyword type as we don't support text push down for now. Issue [BUG][Calcite Engine] Incorrect Pushdown of Equality/Inequality (=, !=) on text Fields #3334 will fix it.

./gradlew :integ-test:integTest --tests '*Calcite*IT' succeed locally except below 3 tests. They failed because of inconsistent order on the final results while it should not promise that order. Need another PR to fix the test themself.

  • CalcitePPLJoinIT.testJoinTwoColumnsAndDisjointFilters
  • CalcitePPLJoinIT.testJoinWithCondition
  • CalcitePPLJoinIT.testJoinWithTwoJoinConditions

Related Issues

Resolves #3368

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Heng Qian <qianheng@amazon.com>
…t type

Signed-off-by: Heng Qian <qianheng@amazon.com>
"type": "keyword"
},
"occupation": {
"type": "text"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@qianheng-aws , please create an issue of supporting text type pushdown. I will add it as child of #3309. Note, the new issue may not require product code change since it could be a part of adding external property to Calcite plan. But at least we need to add some tests to verify the text (may include more) type code be pushdown.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have a issue to fix text push down, #3334

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add this issue as a subtask of that as well, #3370

@LantaoJin
Copy link
Member

Merging to dev branch firstly. cc @penghuo

@LantaoJin LantaoJin merged commit e2d7333 into opensearch-project:feature/calcite-engine Mar 3, 2025
6 of 14 checks passed
penghuo pushed a commit to penghuo/os-sql that referenced this pull request Mar 12, 2025
* Fix PredicateAnalyzer for in and notIn

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Change text field to keyword since we don't support push down for that type

Signed-off-by: Heng Qian <qianheng@amazon.com>

---------

Signed-off-by: Heng Qian <qianheng@amazon.com>
penghuo pushed a commit that referenced this pull request Jun 16, 2025
* Fix PredicateAnalyzer for in and notIn

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Change text field to keyword since we don't support push down for that type

Signed-off-by: Heng Qian <qianheng@amazon.com>

---------

Signed-off-by: Heng Qian <qianheng@amazon.com>
Signed-off-by: xinyual <xinyual@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

calcite calcite migration releated

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants