Skip to content

Make basic aggregation working (part 1) #3318

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Feb 13, 2025
Merged

Make basic aggregation working (part 1) #3318

merged 3 commits into from
Feb 13, 2025

Conversation

@LantaoJin
Copy link
Member

@LantaoJin LantaoJin commented Feb 11, 2025 •
edited
Loading

Description

This PR includes 5 changes:

  1. Introduce OpenSearchTypeSystem (partial code) to address the type mismatch issue in Agg
  2. Using AccessController to workaround the CI failures caused by java security.
  3. Add parameters to enable calcite codegen and debug, ref https://janino-compiler.github.io/janino/#debugging
  4. Refactor Calcite integration tests, now we have both integration test CalcitePPL*IT and unit test CalcitePPL*Test for each command.
  5. Fix bugs in OpenSearchIndexEnumerator

Related Issues

Partially Resolves #3308

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@LantaoJin
Make basic aggregation working (partial)
269187b 
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
add a settings to enable calcite
df413db 
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
add more UTs
d9acec6 
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin LantaoJin marked this pull request as ready for review February 12, 2025 10:31
@LantaoJin
Copy link
Member Author

LantaoJin commented Feb 12, 2025 •
edited
Loading

Ping @qianheng-aws. We may merge PR to dev branch ASAP to unblock peer's tasks. cc @penghuo @dai-chen , you can review and comment regardless of whether PR is merged or not. We will address them in followups.

@LantaoJin LantaoJin changed the title Make basic aggregation working (partial) Make basic aggregation working (part 1) Feb 12, 2025
@penghuo penghuo added the calcite calcite migration releated label Feb 13, 2025
penghuo
penghuo approved these changes Feb 13, 2025
View reviewed changes
core/src/main/java/org/opensearch/sql/executor/OpenSearchTypeSystem.java
return typeFactory.createSqlType(SqlTypeName.DOUBLE);

default:
return super.deriveSumType(typeFactory, argumentType);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why default to deriveSumType instead of deriveAvgAggType?

core/src/main/java/org/opensearch/sql/executor/OpenSearchTypeSystem.java
Comment on lines +21 to +23
case INTEGER:
case BIGINT:
return typeFactory.createSqlType(SqlTypeName.DOUBLE);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it a bug in Calcite? if the argumentType is INTEGER, deriveAvgAggType is INTEGER?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think so, for example table1:

id
1
2
3
4

select avg(id) from table1 returns 2 instead of 2.5.

Not sure it is a bug or by designed. From semantic purpose, query avg on an id column is semantic wrong. The column which could be applied avg should be designed as decimal number. Will open a Calcite issue for checking.

@LantaoJin LantaoJin merged commit e7188da into opensearch-project:feature/calcite-engine Feb 13, 2025
9 of 14 checks passed
@LantaoJin LantaoJin mentioned this pull request Feb 26, 2025
Merged
7 tasks
penghuo pushed a commit to penghuo/os-sql that referenced this pull request Mar 12, 2025
@LantaoJin @penghuo
Make basic aggregation working (part 1) (opensearch-project#3318)
c24b2db 
* Make basic aggregation working (partial)

Signed-off-by: Lantao Jin <ltjin@amazon.com>

* add a settings to enable calcite

Signed-off-by: Lantao Jin <ltjin@amazon.com>

* add more UTs

Signed-off-by: Lantao Jin <ltjin@amazon.com>

---------

Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin LantaoJin mentioned this pull request Apr 11, 2025
Closed
penghuo pushed a commit that referenced this pull request Jun 16, 2025
@LantaoJin @xinyual
Make basic aggregation working (part 1) (#3318)
9000e06 
* Make basic aggregation working (partial)

Signed-off-by: Lantao Jin <ltjin@amazon.com>

* add a settings to enable calcite

Signed-off-by: Lantao Jin <ltjin@amazon.com>

* add more UTs

Signed-off-by: Lantao Jin <ltjin@amazon.com>

---------

Signed-off-by: Lantao Jin <ltjin@amazon.com>
Signed-off-by: xinyual <xinyual@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@penghuo penghuo penghuo approved these changes

@ps48 ps48 Awaiting requested review from ps48 ps48 is a code owner

@kavithacm kavithacm Awaiting requested review from kavithacm kavithacm is a code owner

@derek-ho derek-ho Awaiting requested review from derek-ho derek-ho is a code owner

@joshuali925 joshuali925 Awaiting requested review from joshuali925 joshuali925 is a code owner

@dai-chen dai-chen Awaiting requested review from dai-chen dai-chen is a code owner

@YANG-DB YANG-DB Awaiting requested review from YANG-DB YANG-DB is a code owner

@mengweieric mengweieric Awaiting requested review from mengweieric mengweieric is a code owner

@Swiddis Swiddis Awaiting requested review from Swiddis Swiddis is a code owner

@seankao-az seankao-az Awaiting requested review from seankao-az seankao-az is a code owner

@MaxKsyunz MaxKsyunz Awaiting requested review from MaxKsyunz MaxKsyunz is a code owner

@Yury-Fridlyand Yury-Fridlyand Awaiting requested review from Yury-Fridlyand Yury-Fridlyand is a code owner

@anirudha anirudha Awaiting requested review from anirudha anirudha is a code owner

@forestmvey forestmvey Awaiting requested review from forestmvey forestmvey is a code owner

@acarbonetto acarbonetto Awaiting requested review from acarbonetto acarbonetto is a code owner

@GumpacG GumpacG Awaiting requested review from GumpacG GumpacG is a code owner

@ykmr1224 ykmr1224 Awaiting requested review from ykmr1224 ykmr1224 is a code owner

@noCharger noCharger Awaiting requested review from noCharger noCharger is a code owner

Assignees

No one assigned

Labels

calcite calcite migration releated

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@LantaoJin @penghuo