Skip to content

Implement cryptographic hash UDFs #25

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
yuancu wants to merge 313 commits into from
Closed

Implement cryptographic hash UDFs #25

yuancu wants to merge 313 commits into from

Conversation

@yuancu
Copy link

@yuancu yuancu commented Apr 23, 2025

Description

This PR implements MD5, SHA1, SHA2 with UDFs.

Issues Resolved

opensearch-project#3573

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

penghuo and others added 30 commits June 5, 2024 10:43
@penghuo
Upgrade python 3.8 (opensearch-project#2711)
c90cf00 
Signed-off-by: Peng Huo <penghuo@gmail.com>
@ykmr1224
Add accountId to data models (opensearch-project#2709)
ffc48fa 
Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Revert "Delete Spark datasource (opensearch-project#2638)" (opensearc…
60c4d50 
…h-project#2692)

This reverts commit de7b367.

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@fddattal
penghuo@gmail.com
3f1e3bd 
Signed-off-by: Frank Dattalo <fddattal@amazon.com>
@LantaoJin
Support Percentile in PPL
d767868 
* Support Percentile in PPL

Signed-off-by: Lantao Jin <ltjin@amazon.com>

* Remove ANSI SQL percentile syntax

Signed-off-by: Lantao Jin <ltjin@amazon.com>

* add more unit tests and increase test coverage

Signed-off-by: Lantao Jin <ltjin@amazon.com>

* increase test coverage

Signed-off-by: Lantao Jin <ltjin@amazon.com>

* address comments and add docs

Signed-off-by: Lantao Jin <ltjin@amazon.com>

* add examples in doc

Signed-off-by: Lantao Jin <ltjin@amazon.com>

* fix doctest failure and add more integ tests

Signed-off-by: Lantao Jin <ltjin@amazon.com>

* remove useless code and antlr4 files

Signed-off-by: Lantao Jin <ltjin@amazon.com>

---------

Signed-off-by: Lantao Jin <ltjin@amazon.com>
@ykmr1224
Pass down request context to data accessors (opensearch-project#2715)
c0a5123 
Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Add timeout StatementState (opensearch-project#2724)
3ab7851 
* Add timeout StatementState

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Fix code style

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Fix coverage

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

---------

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@rupal-bq
Add v2.15.0 release notes (opensearch-project#2732)
2aab43d 
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
@ykmr1224
Abstract sessionId generation (opensearch-project#2726)
cd95130 
Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Scaffold async-query-core and async-query module (opensearch-project#…
46015d4 
…2733)

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@seankao31
Handle create index with batch FlintJob (opensearch-project#2734)
b959039 
* update grammar file

Signed-off-by: Sean Kao <seankao@amazon.com>

* batch job for create manual refresh index

Signed-off-by: Sean Kao <seankao@amazon.com>

* dispatcher test for index dml query

Signed-off-by: Sean Kao <seankao@amazon.com>

* borrow lease for refresh query, not batch

Signed-off-by: Sean Kao <seankao@amazon.com>

* spotlessApply

Signed-off-by: Sean Kao <seankao@amazon.com>

* add release note

Signed-off-by: Sean Kao <seankao@amazon.com>

* update comment

Signed-off-by: Sean Kao <seankao@amazon.com>

---------

Signed-off-by: Sean Kao <seankao@amazon.com>
@ykmr1224
Change DataSourceType from enum to class (opensearch-project#2730)
1d703e8 
* Change DataSourceType from enum to class

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Fix test failure

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Fix serialization issue

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Fix format

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Fix integTest

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Fix style

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Fix failing test

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Address comment

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Fix DataSourceType to allow registering new type

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

---------

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Fix code style issue (opensearch-project#2745)
00d5c4e 
Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Move classes from spark to async-query-core and async-query
d5c2fed 
Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Fix build.gradle
61091c1 
Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Adjust build.gradle
ebb07ef 
Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Fix copyrights
084a3c8 
Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Separate build and validateAndBuild method in DataSourceMetadata (ope…
7b40c2c 
…nsearch-project#2744)

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Exclude integ-test, doctest and download task when built offline (ope…
07e52d9 
…nsearch-project#2760)

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Abstract metrics to reduce dependency to legacy (opensearch-project#2747
ef2cef3 
)

* Abstract metrics to reduce dependency to legacy

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Add comment

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Fix style

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

---------

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Remove AsyncQueryId (opensearch-project#2754)
9ad4e02 
Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Add README to async-query-core (opensearch-project#2766)
fbff4a3 
Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Abstract FlintIndex client (opensearch-project#2755)
b2403ca 
* Abstract FlintIndex client

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Fix log

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Fix test function name

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

---------

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Fix statement to store requested langType (opensearch-project#2777)
b9f544b 
Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Push down OpenSearch specific exception handling (opensearch-project#…
8eae36f 
…2778)

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Implement integration test for async-query-core (opensearch-project#2773
49e2e0e 
)

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Fix SQLQueryUtils to extract multiple tables (opensearch-project#2784)
883cc7e 
* Fix SQLQueryUtils to extract multiple tables

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Improve test coverage

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

---------

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Eliminate dependency from async-query-core to legacy (opensearch-proj…
00f82f5 
…ect#2786)

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Pass accountId to EMRServerlessClientFactory.getClient (opensearch-pr…
e24b51f 
…oject#2783)

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Fix build failure (opensearch-project#2794)
4326396 
Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
qianheng-aws and others added 29 commits April 2, 2025 10:48
@qianheng-aws
[Calcite Engine] Support UDT for IP (opensearch-project#3504)
e325431 
* [Calcite Engine] Support UDT for IP

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Fix IT

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Refine Code

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Refine Code

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Address comments

Signed-off-by: Heng Qian <qianheng@amazon.com>

---------

Signed-off-by: Heng Qian <qianheng@amazon.com>
@LantaoJin
Adding new maintainer @qianheng-aws (opensearch-project#3509)
b5dda60 
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Execute Calcite PPL query in thread pool (opensearch-project#3508)
652016c 
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@qianheng-aws
Support GEO_POINT type (opensearch-project#3511)
0754f33 
---------

Signed-off-by: Heng Qian <qianheng@amazon.com>
@xinyual @yuancu
@songkant-aws @qianheng-aws
Add datetime functions (opensearch-project#3473)
920f711 
* add condition and string func

Signed-off-by: xinyual <xinyual@amazon.com>

* add atan

Signed-off-by: xinyual <xinyual@amazon.com>

* [WIP] map opensearch math functions to calcite implementations

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* add transfer for log

Signed-off-by: xinyual <xinyual@amazon.com>

* fix log and atan

Signed-off-by: xinyual <xinyual@amazon.com>

* add udf and udaf percentile and Mod

Signed-off-by: xinyual <xinyual@amazon.com>

* add condition udf and take agg

Signed-off-by: xinyual <xinyual@amazon.com>

* add unified test framework

Signed-off-by: xinyual <xinyual@amazon.com>

* fix UT/IT

Signed-off-by: xinyual <xinyual@amazon.com>

* fix UT/IT

Signed-off-by: xinyual <xinyual@amazon.com>

* remove useless code change

Signed-off-by: xinyual <xinyual@amazon.com>

* apply spotless

Signed-off-by: xinyual <xinyual@amazon.com>

* add license and move it

Signed-off-by: xinyual <xinyual@amazon.com>

* remove useless change

Signed-off-by: xinyual <xinyual@amazon.com>

* add time functions

Signed-off-by: xinyual <xinyual@amazon.com>

* fix date/utc_data/timestamp

Signed-off-by: xinyual <xinyual@amazon.com>

* add unix_timestamp(half)

Signed-off-by: xinyual <xinyual@amazon.com>

* continue to fix unixtimestamp

Signed-off-by: xinyual <xinyual@amazon.com>

* fix unixtimestamp

Signed-off-by: xinyual <xinyual@amazon.com>

* add timestamp

Signed-off-by: xinyual <xinyual@amazon.com>

* fix timestamp

Signed-off-by: xinyual <xinyual@amazon.com>

* Implement DATEADD function

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Implement DATE_SUB function

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* WIP: Implement TIMEADD

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Implement ADDTIME

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Implement DAY_OF_WEEK and DAY_OF_YEAR

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Implement DATEDIFF

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Leverage org.opensearch.sql.utils.DateTimeFormatters for datetime parsing

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Make datetime utils interface

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Implement DATE_FORMAT (can't handle nanoseconds yet)

TODOs:
- support nanoseconds
- correct time to support date as arguments

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Fix rebase

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Fix TIME function
- Additionally apply spotless

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Implement EXTRACT by reusing OS PPL

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Implement CONVERT_TZ. Cannot handle null return at the moment.

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* apply spotless

Signed-off-by: xinyual <xinyual@amazon.com>

* Implement DATETIME
- TODO: handle null returns

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Implement FROM_DAYS

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Implement GET_FORMAT

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Implement MAKETIME
- TODO: fix returned time representation

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* add to days

Signed-off-by: xinyual <xinyual@amazon.com>

* Fix null returns for CONVERT_TZ and DATETIME by manually casting returns to long in UDF

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* add time functions

Signed-off-by: xinyual <xinyual@amazon.com>

* Implement PERIOD_ADD and PERIOD_DIFF

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Implement STR_TO_DATE

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Add license information to datetime UDFs

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Implement SUBTIME

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* add date functions

Signed-off-by: xinyual <xinyual@amazon.com>

* Reimplement WEEK and WEEK_OF_YEAR

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Map CURRENT_TIME to SqlStdOperatorTable.CURRENT_TIME

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Implement MINUTE_OF_DAY

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Fix String.format locale to US for exprMakeTime

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* finish all datetime functions

Signed-off-by: xinyual <xinyual@amazon.com>

* fix implementation

Signed-off-by: xinyual <xinyual@amazon.com>

* add it for time/timestamp/date

Signed-off-by: xinyual <xinyual@amazon.com>

* add relative time

Signed-off-by: xinyual <xinyual@amazon.com>

* add relative

Signed-off-by: xinyual <xinyual@amazon.com>

* fix year bug

Signed-off-by: xinyual <xinyual@amazon.com>

* fix config

Signed-off-by: xinyual <xinyual@amazon.com>

* add several IT

Signed-off-by: xinyual <xinyual@amazon.com>

* Correct DATE_ADD and DATE_SUB implementations

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Fix date and time fields reading

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* fix from unix timestamp

Signed-off-by: xinyual <xinyual@amazon.com>

* add to_days

Signed-off-by: xinyual <xinyual@amazon.com>

* Add date manipulation ITs

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* fix sec to time

Signed-off-by: xinyual <xinyual@amazon.com>

* fix millisecond

Signed-off-by: xinyual <xinyual@amazon.com>

* fix time expression

Signed-off-by: xinyual <xinyual@amazon.com>

* fix nullable problem

Signed-off-by: xinyual <xinyual@amazon.com>

* merge from main

Signed-off-by: xinyual <xinyual@amazon.com>

* Correct DATE_PART related functions (HOUR, MINUTE, etc) to correctly handle string inputs

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Implement MAKEDATE

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Correct periodNameFunction to correctly parse datetime string and to always return English(US)-styled day & month names

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* add IT

Signed-off-by: xinyual <xinyual@amazon.com>

* Support QUARTER function and add ITs

Signed-off-by: Songkan Tang <songkant@amazon.com>

* Add SECOND function ITs

Signed-off-by: Songkan Tang <songkant@amazon.com>

* Add SECOND_OF_MINUTE function ITs

Signed-off-by: Songkan Tang <songkant@amazon.com>

* [Calcite engine] TimeStamp UDT

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Add convert_tz ITs

Signed-off-by: Songkan Tang <songkant@amazon.com>

* Minor fix of get_format function and add ITs

Signed-off-by: Songkan Tang <songkant@amazon.com>

* Correct DATETIME function to accept timestamp argument

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Add extract function with supported ITs

Signed-off-by: Songkan Tang <songkant@amazon.com>

* Correct DATEDIFF to align with V2's behavior

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Correct date part return types & Fix date format locale to English

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Add date time ITs

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Add ITs for SYSDATE and TIMEDIFF

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Revert fixing locale to allow local representation of date time

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* fix sec to time

Signed-off-by: xinyual <xinyual@amazon.com>

* add license

Signed-off-by: xinyual <xinyual@amazon.com>

* finish useless code

Signed-off-by: xinyual <xinyual@amazon.com>

* add IT for percentile

Signed-off-by: xinyual <xinyual@amazon.com>

* rename files

Signed-off-by: xinyual <xinyual@amazon.com>

* fix to days

Signed-off-by: xinyual <xinyual@amazon.com>

* fix

Signed-off-by: xinyual <xinyual@amazon.com>

* apply spotless

Signed-off-by: xinyual <xinyual@amazon.com>

* fix null for yearweek

Signed-off-by: xinyual <xinyual@amazon.com>

* add null IT

Signed-off-by: xinyual <xinyual@amazon.com>

* Add ITs for ADDTIME, ADDDATE, DATESUB, DATEADD, DATE

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Parse date / time exclusively for YEAR, QUARTER, MONTH, DAY, HOUR, MINUTE, SECOND, etc
- it won't parse a time string where a date string is required

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* add IT

Signed-off-by: xinyual <xinyual@amazon.com>

* Support timestamp udt

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Support timestamp udt

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Add date time null input ITs

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* deal invalid cases

Signed-off-by: xinyual <xinyual@amazon.com>

* Make all udf return types nullable

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Remove nullable argument in getReturnTypeForAddOrSubDate
since it's no longer necessary

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* enable nonfallback datetimes

Signed-off-by: xinyual <xinyual@amazon.com>

* Support date, time udt

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Fix date time null returns & Add null ITs

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Allow timestamp string where date or time string is required

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Remove redundant nullify of return types

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* fix percentile

Signed-off-by: xinyual <xinyual@amazon.com>

* Correct makeConversionCall

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Implement TIME UDF

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Fix bug caused by cache in RelDataTypeImpl

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Refine code

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Refine code

Signed-off-by: Heng Qian <qianheng@amazon.com>

* inte with udt

Signed-off-by: xinyual <xinyual@amazon.com>

* Ignore UT with span on calcite date

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Fix bug

Signed-off-by: Heng Qian <qianheng@amazon.com>

* fix add sub time

Signed-off-by: xinyual <xinyual@amazon.com>

* Fix bug

Signed-off-by: Heng Qian <qianheng@amazon.com>

* fix IT

Signed-off-by: xinyual <xinyual@amazon.com>

* fix IT

Signed-off-by: xinyual <xinyual@amazon.com>

* fix return type for str to date

Signed-off-by: xinyual <xinyual@amazon.com>

* fix for subtime

Signed-off-by: xinyual <xinyual@amazon.com>

* add postprocessing for curtimes

Signed-off-by: xinyual <xinyual@amazon.com>

* fix all IT

Signed-off-by: xinyual <xinyual@amazon.com>

* fix format bug

Signed-off-by: xinyual <xinyual@amazon.com>

* fix function properties for all utc function

Signed-off-by: xinyual <xinyual@amazon.com>

* use fixed function properties

Signed-off-by: xinyual <xinyual@amazon.com>

* fix all IT

Signed-off-by: xinyual <xinyual@amazon.com>

* remove all useless code

Signed-off-by: xinyual <xinyual@amazon.com>

* apply spotless

Signed-off-by: xinyual <xinyual@amazon.com>

* Fix date / time parsing with microseconds
- enrich ITs with microseconds

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Test TIMESTAMPADD and TIMESTAMPDIFF with microseconds

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* remove uesless

Signed-off-by: xinyual <xinyual@amazon.com>

* add compare

Signed-off-by: xinyual <xinyual@amazon.com>

* revert change

Signed-off-by: xinyual <xinyual@amazon.com>

* fix IT

Signed-off-by: xinyual <xinyual@amazon.com>

* fix comparison bug

Signed-off-by: xinyual <xinyual@amazon.com>

* fix spotless

Signed-off-by: xinyual <xinyual@amazon.com>

* fix IT

Signed-off-by: xinyual <xinyual@amazon.com>

* fix IT

Signed-off-by: xinyual <xinyual@amazon.com>

* fix IT

Signed-off-by: xinyual <xinyual@amazon.com>

* Fix testMinuteOfHourAndMinuteOfDay IT

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* remove useless code

Signed-off-by: xinyual <xinyual@amazon.com>

* add pushdown IT

Signed-off-by: xinyual <xinyual@amazon.com>

* revert useless change

Signed-off-by: xinyual <xinyual@amazon.com>

* format code

Signed-off-by: xinyual <xinyual@amazon.com>

* modify code

Signed-off-by: xinyual <xinyual@amazon.com>

* remove useless

Signed-off-by: xinyual <xinyual@amazon.com>

* remove useless

Signed-off-by: xinyual <xinyual@amazon.com>

* enable IT

Signed-off-by: xinyual <xinyual@amazon.com>

* apply spotless

Signed-off-by: xinyual <xinyual@amazon.com>

* ignore failed ut

Signed-off-by: xinyual <xinyual@amazon.com>

* use v2 implementation

Signed-off-by: xinyual <xinyual@amazon.com>

* fix implementation

Signed-off-by: xinyual <xinyual@amazon.com>

* use v2 implementation for timestamp and time

Signed-off-by: xinyual <xinyual@amazon.com>

* implement by v2

Signed-off-by: xinyual <xinyual@amazon.com>

* fix implementation

Signed-off-by: xinyual <xinyual@amazon.com>

* fix date format problem

Signed-off-by: xinyual <xinyual@amazon.com>

* remove useless code

Signed-off-by: xinyual <xinyual@amazon.com>

* fix date add/sub

Signed-off-by: xinyual <xinyual@amazon.com>

* remove useless code

Signed-off-by: xinyual <xinyual@amazon.com>

* simplify code

Signed-off-by: xinyual <xinyual@amazon.com>

* refactor date implementation

Signed-off-by: xinyual <xinyual@amazon.com>

* apply spotless

Signed-off-by: xinyual <xinyual@amazon.com>

* add to do

Signed-off-by: xinyual <xinyual@amazon.com>

* add TO DO

Signed-off-by: xinyual <xinyual@amazon.com>

* remove test code

Signed-off-by: xinyual <xinyual@amazon.com>

* refactor code

Signed-off-by: xinyual <xinyual@amazon.com>

* apply spotless

Signed-off-by: xinyual <xinyual@amazon.com>

* fix some bugs

Signed-off-by: xinyual <xinyual@amazon.com>

* fix IT

Signed-off-by: xinyual <xinyual@amazon.com>

* ignore flaky test

Signed-off-by: xinyual <xinyual@amazon.com>

* reformat code to totally use v2 implementation

Signed-off-by: xinyual <xinyual@amazon.com>

* remove useless code

Signed-off-by: xinyual <xinyual@amazon.com>

* remove useless code

Signed-off-by: xinyual <xinyual@amazon.com>

* revert change

Signed-off-by: xinyual <xinyual@amazon.com>

* add copy right

Signed-off-by: xinyual <xinyual@amazon.com>

* change compare to local date time

Signed-off-by: xinyual <xinyual@amazon.com>

* change name after merge

Signed-off-by: xinyual <xinyual@amazon.com>

* remove useless code

Signed-off-by: xinyual <xinyual@amazon.com>

* fix bug

Signed-off-by: xinyual <xinyual@amazon.com>

* ignore flaky test

Signed-off-by: xinyual <xinyual@amazon.com>

* modify IT message information

Signed-off-by: xinyual <xinyual@amazon.com>

* Fix date / time formatting (always prefer Arabic digits instead of local digits)

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* apply spot

Signed-off-by: xinyual <xinyual@amazon.com>

* Reuse PlanUtils.intervalUnitToSpanUnit

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* revert useless code

Signed-off-by: xinyual <xinyual@amazon.com>

* apply spotless and revert useless change

Signed-off-by: xinyual <xinyual@amazon.com>

---------

Signed-off-by: xinyual <xinyual@amazon.com>
Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>
Signed-off-by: Songkan Tang <songkant@amazon.com>
Signed-off-by: Heng Qian <qianheng@amazon.com>
Co-authored-by: xinyual <xinyual@amazon.com>
Co-authored-by: Yuanchun Shen <yuanchu@amazon.com>
Co-authored-by: Songkan Tang <songkant@amazon.com>
Co-authored-by: Heng Qian <qianheng@amazon.com>
@Michael-S
Fix: CSV handling of embedded crlf (opensearch-project#3515)
a039336 
According to RFC 4180 for the CSV file format,
section 2, item 6, if a CSV cell contains a
carriage return ('\r') or line feed ('\n') it
must be quoted.

Signed-off-by: Mike Swierczek <441523+Michael-S@users.noreply.github.com>
@xinyual
use correct timezone name (opensearch-project#3517)
ea020e6
@LantaoJin
Fix antlr4 parser issues (opensearch-project#3492)
d9ef4fa 
* Fix antlr4 parser issues

Signed-off-by: Lantao Jin <ltjin@amazon.com>

* Case insensitive lexer

Signed-off-by: Lantao Jin <ltjin@amazon.com>

---------

Signed-off-by: Lantao Jin <ltjin@amazon.com>
@yuancu
Fix return types of MOD and DIVIDE UDFs (opensearch-project#3513)
b4e8b2e 
* Fix divide function with a UDF

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Coerce return types for DIVIDE and MOD UDFs

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Remove the magic threshold for DIVIDE and MOD
- additionally update expressions.rst to clarify how DIVIDE handles division by zero

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

---------

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>
@LantaoJin
Support line comment and block comment in PPL (opensearch-project#2806)
e0b7268 
* Support line comment and block comment in PPL

Signed-off-by: Lantao Jin <ltjin@amazon.com>

* add user doc

Signed-off-by: Lantao Jin <ltjin@amazon.com>

---------

Signed-off-by: Lantao Jin <ltjin@amazon.com>
@xinyual
fix varchar bug (opensearch-project#3518)
e86f4b6 
* fix varchar bug

Signed-off-by: xinyual <xinyual@amazon.com>

* apply spotless

Signed-off-by: xinyual <xinyual@amazon.com>

* fix wrong IT

Signed-off-by: xinyual <xinyual@amazon.com>

---------

Signed-off-by: xinyual <xinyual@amazon.com>
@qianheng-aws
[Calcite Engine] Function framework refactoring (opensearch-project#3522
b0384a8 
)

---------

Signed-off-by: Heng Qian <qianheng@amazon.com>
@LantaoJin
New output for explain endpoint with Calcite engine (opensearch-proje…
98cee32 
…ct#3521)

---------

Signed-off-by: Lantao Jin <ltjin@amazon.com>
@penghuo
Enable update calcite setting through _plugins/_query/settings API (o…
ab3d8a7 
…pensearch-project#3531)

Signed-off-by: Peng Huo <penghuo@gmail.com>
@noCharger
Add release notes on version 3.0.0.0-beta1 (opensearch-project#3529)
604c923
@pranu2502
Fix build due to phasing off SecurityManager usage in favor of Java A…
4ac9295 
…gent (opensearch-project#3539)

Signed-off-by: Pranav Reddy <pranavrd@amazon.com>
@Siritejagaddameedi
Add SQLQuery Utils support for Vaccum queries (opensearch-project#3269)
e23a61d 
Add UTs



Fix build



Fix code style

Signed-off-by: Siri Teja Gaddameedi <sirigadd@amazon.com>
Co-authored-by: Siri Teja Gaddameedi <sirigadd@amazon.com>
@yuancu
Fix GET_FORMAT UDF (opensearch-project#3543)
d0a6b47 
by adding Locale.ROOT when converting string to lowercase to avoid irregular casting of I when defaulting to Turkic languages

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>
@qianheng-aws
[Calcite Engine] Support alias type field (opensearch-project#3538)
9509c1a 
* [Calcite Engine] Support alias type field

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Add standalone IT to ensure w/wo push down test cases

Signed-off-by: Heng Qian <qianheng@amazon.com>

---------

Signed-off-by: Heng Qian <qianheng@amazon.com>
@pranu2502
Using java-agent gradle plugin to phase off Security Manager in favor…
49a416d 
… of Java-agent. (opensearch-project#3551)

* Revert "Fix build due to phasing off SecurityManager usage in favor of Java Agent (opensearch-project#3539)"

This reverts commit 4ac9295.

Signed-off-by: Pranav Reddy <pranavrd@amazon.com>

* Using java-agent gradle plugin to phase off Security Manager in favor of Java-agent.

Signed-off-by: Pranav Reddy <pranavrd@amazon.com>

---------

Signed-off-by: Pranav Reddy <pranavrd@amazon.com>
@qianheng-aws
[Calcite Engine] Only enable fallback for tests that need to fall back (
3c1591a 
opensearch-project#3544)

* [Calcite Engine] Only enable fallback for tests that need to fall back

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Address comments

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Migrate all calcite remote IT and fix incorrectly ignored test

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Follow the previous action to ignore prometheus test

Signed-off-by: Heng Qian <qianheng@amazon.com>

---------

Signed-off-by: Heng Qian <qianheng@amazon.com>
@qianheng-aws
Support UDT for BINARY (opensearch-project#3549)
15aeec7 
---------

Signed-off-by: Heng Qian <qianheng@amazon.com>
@xinyual
fix timestamp bug (opensearch-project#3542)
eb718fc 
* fix timestamp bug

Signed-off-by: xinyual <xinyual@amazon.com>

* add TODO

Signed-off-by: xinyual <xinyual@amazon.com>

* change doc

Signed-off-by: xinyual <xinyual@amazon.com>

---------

Signed-off-by: xinyual <xinyual@amazon.com>
@penghuo
Fix issue 2489 (opensearch-project#3442)
8134c4e 
* Fix issue 2489

Signed-off-by: Peng Huo <penghuo@gmail.com>

* Update comments

Signed-off-by: Peng Huo <penghuo@gmail.com>

* Integrate with YamlRestTest

Signed-off-by: Peng Huo <penghuo@gmail.com>

* Revert change

Signed-off-by: Peng Huo <penghuo@gmail.com>

* Fix YamlTest

Signed-off-by: Peng Huo <penghuo@gmail.com>

* Disable security.manager

Signed-off-by: Peng Huo <penghuo@gmail.com>

* Remove unused code

Signed-off-by: Peng Huo <penghuo@gmail.com>

---------

Signed-off-by: Peng Huo <penghuo@gmail.com>
@qianheng-aws
[Calcite Engine]Support metadata field (opensearch-project#3445)
9092cc8 
* [Calcite Engine]Support metadata field

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Fix UT/IT

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Refine code

Signed-off-by: Heng Qian <qianheng@amazon.com>

* [Calcite engine] Fix nested subquery

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Fix merging

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Refine Code

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Fix IT

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Address comments

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Address comments

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Address comments3

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Address comments

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Fix Anonymizer

Signed-off-by: Heng Qian <qianheng@amazon.com>

* Address comments

Signed-off-by: Heng Qian <qianheng@amazon.com>

---------

Signed-off-by: Heng Qian <qianheng@amazon.com>
@yuancu
Implement cryptographic functions
616a68c 
Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>
@yuancu
Implement cryptographic UDFs
33c390f 
Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>
@yuancu
Wrap message digests with ThreadLocal to enforce thread safety
f56d5ff 
Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>
@yuancu
Create documentation for cryptographic hash PPL functions
607dab4 
Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>
@yuancu yuancu closed this Apr 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.