[backport to 2.19-dev] Backport calcite prs #3752
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
xinyual
changed the title
xinyual
force-pushed
the
backportForSignoff
branch
from
5fbc9ed to
69763f6
Compare
|
The link checker failure seems due to |
|
…t#3110) Signed-off-by: currantw <taylor.curran@improving.com> Signed-off-by: xinyual <xinyual@amazon.com>
…ject#3075) Signed-off-by: Norman Jordan <norman.jordan@improving.com> Signed-off-by: normanj-bitquill <78755797+normanj-bitquill@users.noreply.github.com> Co-authored-by: Andrew Carbonetto <andrew.carbonetto@improving.com> Signed-off-by: xinyual <xinyual@amazon.com>
* Add trendline (With SWA) PPL command --------- Signed-off-by: James Duong <james.duong@improving.com> Signed-off-by: Andrew Carbonetto <andrew.carbonetto@improving.com> Co-authored-by: Andrew Carbonetto <andrew.carbonetto@improving.com> Signed-off-by: xinyual <xinyual@amazon.com>
Signed-off-by: currantw <taylor.curran@improving.com> * Add support for casting (`cast(field_name to ip)`) and remove existing unused sorting syntax. Signed-off-by: currantw <taylor.curran@improving.com> * Update comparison logic to compare in IPv6 Signed-off-by: currantw <taylor.curran@improving.com> * Fix bug casting to IP Signed-off-by: currantw <taylor.curran@improving.com> * Fix failing tests Signed-off-by: currantw <taylor.curran@improving.com> * Assert that comparison only valid if same type, update tests accordingly Signed-off-by: currantw <taylor.curran@improving.com> * Add additional tests to increase code coverage Signed-off-by: currantw <taylor.curran@improving.com> * Integrate `cidrmatch` changes Signed-off-by: currantw <taylor.curran@improving.com> * Remove `OpenSearchIPType` data type Signed-off-by: currantw <taylor.curran@improving.com> * Fix more failing tests Signed-off-by: currantw <taylor.curran@improving.com> * Minor cleanup Signed-off-by: currantw <taylor.curran@improving.com> * Add new tests for IP data type to `SortCommandIT`, and update `weblogs` test data. Signed-off-by: currantw <taylor.curran@improving.com> * Fixing IT test failure. Signed-off-by: currantw <taylor.curran@improving.com> * Spotless and update test to sort in SQL Signed-off-by: currantw <taylor.curran@improving.com> * Fix broken link Signed-off-by: currantw <taylor.curran@improving.com> * Fix failing code coverage Signed-off-by: currantw <taylor.curran@improving.com> * Fix failing doctest Signed-off-by: currantw <taylor.curran@improving.com> * Fix failing `ip.rst` doctest Signed-off-by: currantw <taylor.curran@improving.com> * Fix test failure due to merge. Signed-off-by: currantw <taylor.curran@improving.com> * Fix spotless Signed-off-by: currantw <taylor.curran@improving.com> * Add missing `url` field Signed-off-by: currantw <taylor.curran@improving.com> * Address minor review comments. Signed-off-by: currantw <taylor.curran@improving.com> * Revert sort syntax changes Signed-off-by: currantw <taylor.curran@improving.com> * Minor doc update Signed-off-by: currantw <taylor.curran@improving.com> * FIx failing `ip.rst` doctest Signed-off-by: currantw <taylor.curran@improving.com> * Add `IPComparisonIT` tests for comparison operators, rename modules and weblogs test index to make plural for consistency. Signed-off-by: currantw <taylor.curran@improving.com> --------- Signed-off-by: currantw <taylor.curran@improving.com> Signed-off-by: xinyual <xinyual@amazon.com>
…er (opensearch-project#3249) * First commit for Calcite integration Signed-off-by: Lantao Jin <ltjin@amazon.com> * disable java security manager in IT Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: xinyual <xinyual@amazon.com>
…ject#3258) * [POC] Make Calcite execute successfully Signed-off-by: Heng Qian <qianheng@amazon.com> * [POC] Change caching schema to simple schema and avoid registering table when visitRelation. Signed-off-by: Heng Qian <qianheng@amazon.com> * spotlessApply Signed-off-by: Heng Qian <qianheng@amazon.com> * address comments Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: xinyual <xinyual@amazon.com>
* Make basic aggregation working (partial) Signed-off-by: Lantao Jin <ltjin@amazon.com> * add a settings to enable calcite Signed-off-by: Lantao Jin <ltjin@amazon.com> * add more UTs Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: xinyual <xinyual@amazon.com>
opensearch-project#3327) * Support Filter and Project pushdown Signed-off-by: Heng Qian <qianheng@amazon.com> * Support Filter and Project pushdown v2 Signed-off-by: Heng Qian <qianheng@amazon.com> * Address comments Signed-off-by: Heng Qian <qianheng@amazon.com> * Add original license for PredicateAnalyzer Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: xinyual <xinyual@amazon.com>
* Build integration test framework Signed-off-by: Lantao Jin <ltjin@amazon.com> * make local work Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix the timestamp issue Signed-off-by: Lantao Jin <ltjin@amazon.com> * address comments Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix java style and rename CalcitePPLTestCase back to CalcitePPLIntegTestCase Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: xinyual <xinyual@amazon.com>
…nsearch-project#3349) Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: xinyual <xinyual@amazon.com>
…oject#3355) * Add more aggregation tests Signed-off-by: Lantao Jin <ltjin@amazon.com> * delete unrelavant code Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: xinyual <xinyual@amazon.com>
* Transform to calcite plan before executing Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix bug for single column row Signed-off-by: Heng Qian <qianheng@amazon.com> * Add settings for calcite pushdown Signed-off-by: Heng Qian <qianheng@amazon.com> * Lazily construct OpenSearchRequestBuilder and do push down Signed-off-by: Heng Qian <qianheng@amazon.com> * Address comments and disable push down Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: xinyual <xinyual@amazon.com>
Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: xinyual <xinyual@amazon.com>
Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: xinyual <xinyual@amazon.com>
* Fix PredicateAnalyzer for in and notIn Signed-off-by: Heng Qian <qianheng@amazon.com> * Change text field to keyword since we don't support push down for that type Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: xinyual <xinyual@amazon.com>
…3371) Signed-off-by: xinyual <xinyual@amazon.com>
Signed-off-by: xinyual <xinyual@amazon.com>
Signed-off-by: xinyual <xinyual@amazon.com>
…3376) * [BugFix] Fix text field push down Signed-off-by: Heng Qian <qianheng@amazon.com> * Ignore CalciteSortCommandIT.testSortWithNullValue Signed-off-by: Heng Qian <qianheng@amazon.com> * Refine code: only get keyword subfield for termQuery builder Signed-off-by: Heng Qian <qianheng@amazon.com> * Refine code Signed-off-by: Heng Qian <qianheng@amazon.com> * remove ignore tests in CalcitePPLInSubqueryIT Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: xinyual <xinyual@amazon.com>
* add udf/udaf interface and take/sqrt function Signed-off-by: xinyual <xinyual@amazon.com> * add UT Signed-off-by: xinyual <xinyual@amazon.com> * add POW, Atan, Atan2 and corresponding UT Signed-off-by: xinyual <xinyual@amazon.com> * apply spotless Signed-off-by: xinyual <xinyual@amazon.com> * fix table for join it Signed-off-by: xinyual <xinyual@amazon.com> * add java doc Signed-off-by: xinyual <xinyual@amazon.com> * apply spotless Signed-off-by: xinyual <xinyual@amazon.com> --------- Signed-off-by: xinyual <xinyual@amazon.com>
…ry impl (opensearch-project#3377) Signed-off-by: xinyual <xinyual@amazon.com>
…ect#3378) Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: xinyual <xinyual@amazon.com>
…t#3388) Signed-off-by: xinyual <xinyual@amazon.com>
…t#3392) * Implement ppl scalar subquery command with Calcite Signed-off-by: Lantao Jin <ltjin@amazon.com> * more general subquery checker Signed-off-by: Lantao Jin <ltjin@amazon.com> * support correlated IN subquery Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: xinyual <xinyual@amazon.com>
* Change push down to logical index scan Signed-off-by: Heng Qian <qianheng@amazon.com> * Support Aggregate Push Down Signed-off-by: Heng Qian <qianheng@amazon.com> * Rebase and resolve conflict Signed-off-by: Heng Qian <qianheng@amazon.com> * Add TODO Signed-off-by: Heng Qian <qianheng@amazon.com> * Address comments Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: xinyual <xinyual@amazon.com>
* add string udfs Signed-off-by: xinyual <xinyual@amazon.com> * add it to string Signed-off-by: xinyual <xinyual@amazon.com> * add IT for string function Signed-off-by: xinyual <xinyual@amazon.com> * remove change for local test Signed-off-by: xinyual <xinyual@amazon.com> * revert change Signed-off-by: xinyual <xinyual@amazon.com> --------- Signed-off-by: xinyual <xinyual@amazon.com>
Signed-off-by: xinyual <xinyual@amazon.com>
LantaoJin
requested review from
GumpacG,
MaxKsyunz,
Swiddis,
YANG-DB,
Yury-Fridlyand,
acarbonetto,
anirudha,
dai-chen,
derek-ho,
forestmvey,
joshuali925,
kavithacm,
mengweieric,
penghuo,
ps48,
seankao-az and
ykmr1224
as code owners
LantaoJin
previously approved these changes
Jun 12, 2025
|
@penghuo please double confirm the "?" item in #3752 (comment) and merge it. |
#3715, not required. |
Signed-off-by: xinyual <xinyual@amazon.com>
|
@qianheng-aws Please review it. |
|
@noCharger Do we need to backport async-query-core to 2.19-dev? 2 related PRs |
It's good to have them on 2.19 to benefit people using 2.19 cluster. Not must-have though. |
penghuo
approved these changes
Jun 13, 2025
LantaoJin
approved these changes
Jun 16, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The PR backport all calcite feature PRs and some pre-required PR to 2.19-dev.
The PR is based on 2.19.0-SNAPSHOT and delete maven-publish to avoid to publish to maven repo.
Some dependencies' version are fixed from core
Backporting all PRs in main branch Starting from #3249 to now.
It also includes some PRs not related to calcite but as pre-required to simplify conflict resolve:
#3036
#3032
#3071
#3145
#3230
#3246
Remaining PR from #3249 not backported
#3256 CI: Fix broken url for link-checker CI build
#3254 Unused PPL EARLIEST and LATEST aggregations
#3255 CI: BWC testsuite
#3244 CI: Geo-Spatial Plugin integration test (need for geo IP)
#3259 CI: Syntax fix for Jekyll build
#3228 PPL command expression implementation for geoip(need for geo ip)
#3274 Update PPL parse command doc with limitation
#3299 Fixed week of week based year handling
#3300 [Release 2.19] Add release notes for 2.19
#3294 Build: Centralise dependencies version - Pt1
#3304 Add other functions to SQL query validator
#3306 [v3.0.0] Remove SparkSQL support
#3243 PPL: Add json function and cast(x as json) function
#2891 Remove dependency from async-query-core to datasources
#3326 Remove opendistro settings and endpoints
#3319 [Release 3.0] Bump gradle 8.10.2 / JDK23 / 3.0.0.0-alpha1 on SQL plugin
#3337 [v3.0.0] Deprecate SQL Delete statement
#3278 Clean up syntax error reporting
#3367 [v3.0.0] Deprecate OpenSearch DSL format
#3414 Add release notes on version 3.0.0.0-alpha1
#3434 Merge main for OpenSearch 3.0 release.
#3469 Set bouncycastle version inline
#3484 CVE-2024-57699 High: Fix json-smart vulnerability
#3489 Bump SQL main to version 3.0.0.0-beta1
#3515 Fix: CSV handling of embedded crlf
#3539 Fix build due to phasing off SecurityManager usage in favor of Java Agent
#3269 Add SQLQuery Utils support for Vaccum queries
#3551 Using java-agent gradle plugin to phase off Security M anager in favor of Java-agent.
#3442 Fix issue 2489 [backported]
#3649 create a new directory org/opensearch/direct-query/
#3671 [Fix CVE-2025-47273]Bump setuptools to 78.1.1 [backported]
#3604 Implement geoip udf with Calcite (need for geo ip) [backported]
#3660 Fix: Long IN-lists causes crash [backported]
#3715 Modified workflow: Grammar Files & Async Query Core
#3588 [AUTO] Increment version to 3.1.0-SNAPSHOT
#3589 Remove beta1 qualifier
#3592 Add release notes on version 3.0.0.0
The difference in functionality between this PR and main:
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
--signoff.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.