Implement half of date / time functions #7

yuancu · 2025-03-17T10:40:44Z

Description

[Describe what this change achieves]

Issues Resolved

[List any issues this PR will resolve]

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…error (opensearch-project#2626) Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>

Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>

Instead of downloading the JAR from Maven, the JAR in the EMR serverless root file system can be used. Signed-off-by: Adi Suresh <adsuresh@amazon.com>

…oject#2624) * Add option to use LakeFormation in S3Glue data source. Signed-off-by: Adi Suresh <adsuresh@amazon.com> * Update s3glue_connector.rst corrected formatting issue. Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> --------- Signed-off-by: Adi Suresh <adsuresh@amazon.com> Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> Co-authored-by: Vamsi Manohar <reddyvam@amazon.com>

* Delete Spark datasource Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Fix build error Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Delete spark_connector.rst Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Add missing test Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Refactor SparkQueryDispatcher Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Remove EMRServerlessClientFactory from SparkQueryDispatcher Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Fix unit test failures in SparkQueryDispatcherTest Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

…-project#2645) Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

Signed-off-by: Sean Kao <seankao@amazon.com>

* Introduce FlintIndexStateModelService Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Reformat Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Add comments to query handlers Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Reformat Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Fix comments Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

Signed-off-by: varun-lodaya <varunml@amazon.com>

* Bump bwc version to 2.15 Signed-off-by: Derek Ho <dxho@amazon.com> * Skip macos Signed-off-by: Derek Ho <dxho@amazon.com> --------- Signed-off-by: Derek Ho <dxho@amazon.com>

…-project#2665) * Extract SessionStorageService and StatementStorageService Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Reformat Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Add copyright comment Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Add comments and remove unused methods Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Remove unneeded imports Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Fix code format issue Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

…t#2687) * Refactor data models to be generic to data storage Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Address review comments Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Reduce redundancy Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Provide a way to modify spark parameters Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Address review comment Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Address review comment Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Address review comment Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

…#2693) * Change JobExecutionResponseReader to an interface Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Fix comment Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Abstract queryId generation Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Remove OpenSearch specific id mapping from model classes Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Fix code style Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

…ect#2707) Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

Signed-off-by: Peng Huo <penghuo@gmail.com>

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

…h-project#2692) This reverts commit de7b367. Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

Signed-off-by: Frank Dattalo <fddattal@amazon.com>

* Support Percentile in PPL Signed-off-by: Lantao Jin <ltjin@amazon.com> * Remove ANSI SQL percentile syntax Signed-off-by: Lantao Jin <ltjin@amazon.com> * add more unit tests and increase test coverage Signed-off-by: Lantao Jin <ltjin@amazon.com> * increase test coverage Signed-off-by: Lantao Jin <ltjin@amazon.com> * address comments and add docs Signed-off-by: Lantao Jin <ltjin@amazon.com> * add examples in doc Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix doctest failure and add more integ tests Signed-off-by: Lantao Jin <ltjin@amazon.com> * remove useless code and antlr4 files Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com>

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Add timeout StatementState Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Fix code style Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Fix coverage Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

…sing Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

TODOs: - support nanoseconds - correct time to support date as arguments Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* revert result ordering of stats-by Signed-off-by: Lantao Jin <ltjin@amazon.com> * Fix CRLF issue Signed-off-by: Lantao Jin <ltjin@amazon.com> * only check spark sql Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com>

WIP: Implement datetime functions

- Additionally apply spotless Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

…unctions

Fix TIME issues

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Signed-off-by: xinyual <xinyual@amazon.com>

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

- TODO: handle null returns Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

- TODO: fix returned time representation Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

…rns to long in UDF Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

vamsimanohar and others added 30 commits April 22, 2024 13:15

Throw OpensearchSecurityException incase of datasource authorization …

5464bfc

…error (opensearch-project#2626) Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>

Bump bwc version (opensearch-project#2616)

2c97be7

Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>

Use EMR serverless bundled iceberg JAR. (opensearch-project#2632)

e578a57

Instead of downloading the JAR from Maven, the JAR in the EMR serverless root file system can be used. Signed-off-by: Adi Suresh <adsuresh@amazon.com>

Fix test failure in SparkQueryDispatcherTest due to merge (opensearch…

0f4e1c0

…-project#2645) Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

release notes for 2.14.0.0 (opensearch-project#2647)

b454a2c

Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>

Refactor IndexDMLHandler and related classes (opensearch-project#2644)

45122ec

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

Update maintainers list (opensearch-project#2663)

cfd222c

Signed-off-by: Sean Kao <seankao@amazon.com>

Introduce FlintIndexStateModelService (opensearch-project#2658)

df1c04a

* Introduce FlintIndexStateModelService Signed-off-by: Tomoyuki Morita <moritato@amazon.com> * Reformat Signed-off-by: Tomoyuki Morita <moritato@amazon.com> --------- Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

Updating security reachout email (opensearch-project#2671)

58c55e7

Signed-off-by: varun-lodaya <varunml@amazon.com>

Bump bwc version to 2.15 (opensearch-project#2662)

8f06b6f

* Bump bwc version to 2.15 Signed-off-by: Derek Ho <dxho@amazon.com> * Skip macos Signed-off-by: Derek Ho <dxho@amazon.com> --------- Signed-off-by: Derek Ho <dxho@amazon.com>

Make models free of XContent

afec315

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

Add comments

2c849f7

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

Remove unneeded datasourceName parameters (opensearch-project#2683)

a64fcb1

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

Introduce SessionConfigSupplier to abstract settings (opensearch-proj…

65e88c2

…ect#2707) Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

Upgrade python 3.8 (opensearch-project#2711)

c90cf00

Signed-off-by: Peng Huo <penghuo@gmail.com>

Add accountId to data models (opensearch-project#2709)

ffc48fa

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

Revert "Delete Spark datasource (opensearch-project#2638)" (opensearc…

60c4d50

…h-project#2692) This reverts commit de7b367. Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

penghuo@gmail.com

3f1e3bd

Signed-off-by: Frank Dattalo <fddattal@amazon.com>

Pass down request context to data accessors (opensearch-project#2715)

c0a5123

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

yuancu and others added 29 commits March 14, 2025 16:16

Implement DAY_OF_WEEK and DAY_OF_YEAR

02be4c4

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Implement DATEDIFF

6ca160d

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Leverage org.opensearch.sql.utils.DateTimeFormatters for datetime par…

a359112

…sing Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Make datetime utils interface

0e8e5be

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Implement DATE_FORMAT (can't handle nanoseconds yet)

8338a58

TODOs: - support nanoseconds - correct time to support date as arguments Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Merge pull request #4 from yuancu/addTimeFunctions

6849035

WIP: Implement datetime functions

Fix rebase

cbd95d6

Fix TIME function

345de93

- Additionally apply spotless Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Merge remote-tracking branch 'xinyuan/addTimeFunctions' into addTimeF…

e5a347d

…unctions

Merge pull request #5 from yuancu/addTimeFunctions

03e0726

Fix TIME issues

Implement EXTRACT by reusing OS PPL

b8a8667

Implement CONVERT_TZ. Cannot handle null return at the moment.

c78f6ca

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

merge from main

d8201da

Signed-off-by: xinyual <xinyual@amazon.com>

apply spotless

5f72f21

Signed-off-by: xinyual <xinyual@amazon.com>

Merge branch 'addTimeFunctions' into cleanDateTime

40d4caa

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Implement DATETIME

2962942

- TODO: handle null returns Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Implement FROM_DAYS

c94657c

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Implement GET_FORMAT

6711b82

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Implement MAKETIME

b8509d2

- TODO: fix returned time representation Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Fix null returns for CONVERT_TZ and DATETIME by manually casting retu…

c3eb8ff

…rns to long in UDF Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Implement PERIOD_ADD and PERIOD_DIFF

ee9f704

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Implement STR_TO_DATE

7a8674d

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Add license information to datetime UDFs

40b166c

Implement SUBTIME

6f3545a

Reimplement WEEK and WEEK_OF_YEAR

ae42653

Map CURRENT_TIME to SqlStdOperatorTable.CURRENT_TIME

8e04bde

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Implement MINUTE_OF_DAY

8a45c36

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Fix String.format locale to US for exprMakeTime

9db5a08

yuancu closed this Mar 17, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement half of date / time functions #7

Implement half of date / time functions #7

Uh oh!

yuancu commented Mar 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Implement half of date / time functions #7

Implement half of date / time functions #7

Uh oh!

Conversation

yuancu commented Mar 17, 2025

Description

Issues Resolved

Check List

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants