2.12.0.0

What's Changed

• [Backport 2.x] Add release notes for 2.2.0.0 by @opensearch-trigger-bot in #731
• Backport Spring and jackson-databind version bump to 2.x by @dai-chen in #759
• [Backport 2.x] Replace 2x LogUtils by QueryContext by @opensearch-trigger-bot in #753
• [Backport 2.x] Update snapshots for jest tests on workbench by @opensearch-trigger-bot in #761
• Merge main to 2.x branch by @joshuali925 in #771
• [Backport 2.x] Bump Checkstyle version to latest by @opensearch-trigger-bot in #776
• [Backport 2.x] Change master node timeout to new API by @opensearch-trigger-bot in #797
• Bugfix, copy of AggregationOperator should be same by @penghuo in #806
• Add Forum link in SQL plugin README.md by @cwillum in #809
• Catalog Implementation by @vamsi-amazon in #819
• add 2.3 release notes by @penghuo in #824
• Fix failing ODBC workflow by @Yury-Fridlyand in #828
• Merge main to 2.x by @penghuo in #830
• Bump version to 2.4 by @penghuo in #832
• Change ApplicationContext lifecycle to per node level by @penghuo in #822
• Fix EqualsAndHashCode Annotation Warning Messages by @GabeFernandez310 in #847
• Add low-level create table and table exists API by @dai-chen in #834
• Add implementation of now, sysdate, localtime and similar functions by @Yury-Fridlyand in #754
• Update com.fasterxml.jackson to 2.13.4 to match opensearch repo. by @MaxKsyunz in #858
• Add datetime functions FROM_UNIXTIME and UNIX_TIMESTAMP by @Yury-Fridlyand in #835
• Reorganize GitHub workflows. by @Yury-Fridlyand in #837
• Adding CONVERT_TZ and DATETIME functions to SQL and PPL by @MitchellGale in #848
• Remove duplicated png file by @joshuali925 in #865
• Update Jetty Dependency by @forestmvey in #872
• Trigger build on pull request synchronize action. by @vamsi-amazon in #873
• Add patterns and grok command by @joshuali925 in #813
• Fix NPE with multiple queries containing DOT(.) in index name. by @vamsi-amazon in #870
• Fix indentation of patterns example by @joshuali925 in #880
• Prometheus Connector Initial Code by @vamsi-amazon in #878
• Restricted catalog name to [a-zA-Z0-9_-] characters by @vamsi-amazon in #876
• Update docs - missing changes for #754. by @Yury-Fridlyand in #884
• Add Support for Highlight Wildcard in SQL by @forestmvey in #827
• add groupId to pluginzip publication by @penghuo in #906
• Update group to groupId by @penghuo in #908
• Fix broken links by @Yury-Fridlyand in #911
• Update SQL CLI to use AWS session token. by @Yury-Fridlyand in #918
• Table function for supporting prometheus query_range function by @vamsi-amazon in #875
• Update version of jackson-databind for sql-jdbc only by @GumpacG in #943
• Add security policy for ml-commons library by @joshuali925 in #945
• Add typeof function. by @Yury-Fridlyand in #867
• Update JDBC driver version by @GumpacG in #941
• Fix result order of parse with other run time fields by @joshuali925 in #934
• Show catalogs by @vamsi-amazon in #925
• feat: enable ci for windows and macos by @derek-ho in #907
• change condition to always upload coverage for linux workbench by @derek-ho in #967
• Add category_field to AD command in PPL by @joshuali925 in #952
• Adding docs related to catalog. by @vamsi-amazon in #963
• AD timefield name issue by @jngz-es in #919
• Bump ansi-regex for workbench by @joshuali925 in #975
• Fix manual CI workflow and add name option. by @Yury-Fridlyand in #904
• Removed json-smart in the JDBC driver by @GumpacG in #978
• Add time window and window assigner by @dai-chen in #950
• Add take() aggregation function in PPL by @joshuali925 in #949
• Merge 2.x to feature/maximus-m1 branch by @penghuo in #979
• SHOW CATALOGS documentation and integ tests by @vamsi-amazon in #977
• Add metadatalog interface and default in memory implementation by @penghuo in #974
• Add functions PERIOD_ADD and PERIOD_DIFF. by @Yury-Fridlyand in #933
• Enable ignored and disabled tests by @GumpacG in #926
• List tables/metrics using information_schema in source command. by @vamsi-amazon in #914
• Update MacOS Version for ODBC Driver by @forestmvey in #987
• Add Statement, QueryExecution and QueryManager by @penghuo in #845
• Update Jackson Databind version to 2.13.4.2 by @margarit-h in #992
• A Generic ML Command in PPL by @jngz-es in #971
• Describe Table with catalog name. by @vamsi-amazon in #989
• Catalog Enhancements by @vamsi-amazon in #988
• Rework on error reporting to make it more verbose and human-friendly. by @Yury-Fridlyand in #839
• Add watermark generator by @dai-chen in #959
• Add stream context and window trigger by @dai-chen in #958
• Add document for ml command. by @jngz-es in #1001
• Add function name as identifier in antlr by @penghuo in #1015
• Disable loading ML-commons plugin for doctests by @Yury-Fridlyand in #1011
• Prometheus select metric and stats queries. by @vamsi-amazon in #956
• Fix incorrect results returned by min, max and avg by @Yury-Fridlyand in #1000
• Bump sql-cli version to 1.1.0 by @penghuo in #1023
• [Backport 2.x] Add 2.4 release note for SQL/PPL by @opensearch-trigger-bot in #1034
• Fix FLOAT -> DOUBLE cast. by @Yury-Fridlyand in #1025
• Merge 2.x to Maximus feature branch by @dai-chen in #1039
• [Backport 2.x] Fix error messaging from prometheus. by @opensearch-trigger-bot in #1037
• Add Query Function As Alternate Syntax to Query_string Function by @GabeFernandez310 in #1010
• Deprecate span collector by @dai-chen in #990
• Update DATE and TIME functions to parse string input as datetime by @MitchellGale in #991
• Add Streaming source impl by @penghuo in #994
• [Backport 2.x] Fix for errors in prometheus connector when backquotes are used for fieldnames in aggregations. by @opensearch-trigger-bot in #1050
• [Backport to 2.x]Catalog to Datasource changes (#1027) by @vamsi-amazon in #1049
• [Backport 2.x] [2.4] Bump jackson to 2.14.0 by @opensearch-trigger-bot in #1060
• Add valueOf() to Expression by @joshuali925 in #1055
• M...

Version 2.9.0

Compatible with OpenSearch and OpenSearch Dashboards Version 2.9.0

Features

• Enable Table Function and PromQL function (#1719)
• Add Spark connector (#1780)

Enhancements

• Pagination: Support WHERE clause, column list in SELECT clause and for functions and expressions in the query (#1500)
• Pagination: Support ORDER BY clauses and queries without FROM clause (#1599)
• Remove backticks on by field in stats (#1728)
• Support Array and ExprValue Parsing With Inner Hits (#1737)
• Add Support for Nested Function in Order By Clause (#1789)
• Add Support for Field Star in Nested Function (#1773)
• Guarantee datasource read api is strong consistent read (compatibility with segment replication) (#1815)
• Added new datetime functions and aliases to PPL (#1807)
• Support user-defined and incomplete date formats (#1821)
• Add _routing to SQL includes list (#1771)
• Disable read of plugins.query.datasources.encryption.masterkey from cluster settings GET API (#1825)
• Add EMR client to spark connector (#1790)
• Improved error codes in case of data source API security exception (#1753)
• Remove Default master encryption key from settings (#1851)

Bug Fixes

• Fixed bug of byte/short not handling 0 denominator in divide/modulus equations (#1716)
• Fix CSV/RAW output header being application/json rather than plain/text (#1779)

Documentation

• Updated documentation of round function return type (#1725)
• Updated protocol.rst with new wording for error message (#1662)
• Updated documentation for temporal data types (#1826)

Infrastructure

• stopPrometheus task in doctest build.gradle now runs upon project failure in startOpenSearch (#1747)
• Bump guava to 32.0.1 (#1829)
• Disable CrossClusterSearchIT test (#1814)
• Fix flakytest when tests.locale=tr (#1827)

Refactoring

• Simplify OpenSearchIndexScanBuilder (#1738)

Version 2.8.0

Compatible with OpenSearch and OpenSearch Dashboards Version 2.8.0

Release date: June 6th, 2023

• Support for pagination in v2 engine of SELECT FROM table queries (#1666)
• Support Alternate Datetime Formats (#1664)
• Create new anonymizer for new engine (#1665)
• Add Support for Nested Function Use In WHERE Clause Predicate Expresion (#1657)
• Cross cluster search in PPL (#1512)
• Added COSH to V2 engine (#1428)
• REST API for GET,PUT and DELETE (#1482)

Enhancements

• Minor clean up of datetime and other classes (#1310)
• Add integration JDBC tests for cursor/fetch_size feature (#1315)
• Refactoring datasource changes to a new module. (#1504)

Bug Fixes

• Fixing bug where Nested functions used in WHERE, GROUP BY, HAVING, and ORDER BY clauses don't fallback to legacy engine. (#1549)

Documentation

• Add Nested Documentation for 2.7 Related Features (#1620)
• Update usage example doc for PPL cross-cluster search (#1610)
• Documentation and other papercuts for datasource api launch (#1530)

Maintenance

• Fix IT - address breaking changes from upstream. (#1659)
• Increment version to 2.8.0-SNAPSHOT (#1552)
• Backport maintainer list update to 2.x. (#1650)
• Backport jackson and gradle update from #1580 to 2.x (#1596)
• adding reflections as a dependency (#1559)
• Bump org.json dependency version (#1586)
• Integ Test Fix (#1541)

Version 2.7.0

What's Changed

• Add release notes for 2.6.0 by @rupal-bq in #1369
• Integ Test Refactoring by @vamsi-amazon in #1383
• Refactor AWSSigV4 auth to support different AWSCredentialProviders by @vamsi-amazon in #1389
• Exclude OpenSearch system index when IT cleanup by @dai-chen in #1381
• [AUTO] Increment version to 2.7.0-SNAPSHOT by @opensearch-trigger-bot in #1368
• Update usage of Strings.toString by @MaxKsyunz in #1404
• Deprecated Spring IoC and using Guice instead (#1177) by @penghuo in #1410
• [main]Changes in DataSourceService and DataSourceMetadataStorage interface by @vamsi-amazon in #1414
• Bump backport version. by @Yury-Fridlyand in #1009
• Resolve table function based on StorageEngine provided function resolver by @penghuo in #1424
• Added SINH function to V2 engine by @matthewryanwells in #1437
• Added RINT function to V2 engine by @matthewryanwells in #1439
• Add sec_to_time Function To OpenSearch SQL by @GabeFernandez310 in #1438
• Add WEEKDAY Function to SQL Plugin by @GabeFernandez310 in #1440
• Add YEARWEEK Function To OpenSearch SQL by @GabeFernandez310 in #1445
• Add EXTRACT Function To OpenSearch SQL Plugin by @GabeFernandez310 in #1443
• Add STR_TO_DATE Function To The SQL Plugin by @GabeFernandez310 in #1444
• Add The TO_SECONDS Function To The SQL Plugin by @GabeFernandez310 in #1447
• Added Arithmetic functions to V2 engine by @matthewryanwells in #1448
• Added SIGNUM function to V2 engine by @matthewryanwells in #1442
• Add TIMESTAMPADD Function To OpenSearch SQL Plugin by @GabeFernandez310 in #1453
• Rework on OpenSearchDataType: parse, store and use mapping information by @Yury-Fridlyand in #1455
• Fix null response from pow/power and added missing integration testing by @matthewryanwells in #1459
• Add Timestampdiff Function To OpenSearch SQL by @GabeFernandez310 in #1472
• Create datasource API (#1458) by @vamsi-amazon in #1479
• Update to account for XContent refactor in 2.x by @MaxKsyunz in #1485
• Replace non-ASCII characters in code and docs. by @Yury-Fridlyand in #1486
• REST API for GET,PUT and DELETE by @vamsi-amazon in #1502
• #639: allow metadata fields and score opensearch function by @acarbonetto in #1509
• Add Nested Support in Select Clause (#1490) by @forestmvey in #1518
• Refactoring datasource changes to a new module. by @vamsi-amazon in #1511
• Documentation and other papercuts for datasource api launch by @vamsi-amazon in #1534
• Release Notes for 2.7.0 by @vamsi-amazon in #1540
• Datasources Integ Test Fix by @vamsi-amazon in #1544
• release notes for 2.7.0 PATCH by @forestmvey in #1557
• Ensure Nested Function Falls Back to Legacy Engine Where Not Supported by @forestmvey in #1555
• adding reflections as a dependency by @dhrubo-os in #1561
• updating release notes 2.7.0.0 by @dhrubo-os in #1562
• Bump org.json dependency version by @dai-chen in #1586

New Contributors

• @dhrubo-os made their first contribution in #1561

Full Changelog: 2.6.0.0...2.7.0.0

Version 2.5.0

Compatible with OpenSearch and OpenSearch Dashboards Version 2.5.0

Refactoring

• The SQL Plugin was rearranged into separate repositories (#640 #1263)
• Added SQL-Jdbc repository https://github.com/opensearch-project/sql-jdbc (#964)
• Added SQL-Jdbc maven artifact as a dependency (#2692)
• Added SQL-Odbc repository https://github.com/opensearch-project/sql-odbc (#965)
• Added SQL-Cli repository https://github.com/opensearch-project/sql-cli (#966)
• Added Workbench repository https://github.com/opensearch-project/dashboards-query-workbench (#1266)

Enhancements

• Add low-level create table and table exists API (#834)
• Add time window and window assigner (#950)
• Add valueOf() to Expression (#1055)
• Add Statement, QueryExecution and QueryManager (#845)
• Add Streaming Source Impl (#994)
• Add watermark generator (#959)
• Add stream context and window trigger (#958)
• Add micro batch streaming execution (#1044)
• Add Streaming Plan Impl (#1068)
• Add CBRT to the V2 engine (#1081)
• Add CBRT function to the PPL (#1097)
• Add timeout option to SQL CLI tool. (#1076)
• Add Day_Of_Year Function To OpenSearch (#1128)
• Add Week_Of_Year Function To OpenSearch (#1127)
• Add Month_Of_Year Function To OpenSearch (#1129)
• Add Minute_Of_Day Function To SQL Plugin (#1207 #1214)
• Add Second_Of_Minute Function As An Alias Of The Second Function (#1231 #1237)
• Add Support For Legacy Syntax For Match Function In New Engine (#1090)
• Add MatchPhraseQuery As Alternate Syntax for Match_Phrase Function (#1103)
• Use query execution start time as the value of now-like functions. (#1047)
• Add Support for Alternate Legacy MULTIMATCH syntax (#1102)
• Add position() function to V2 engine (#1121)
• Add position() string function to PPL (#1147)
• Add support for wildcard_query function to the new engine (#156 #1108)
• Add reverse() string function to V2 SQL Engine(#1154)
• Add table write operator and builder (#1094)
• Add BETWEEN expression in v2 engine (#1163)
• Adding UTC_DATE, UTC_TIME, UTC_TIMESTAMP (#1193 #1198)
• Validate field and fields parameters in relevance search functions (#1067 #1199)
• Add TIMEDIFF and DATEDIFF functions. (#131 #1195 #1234)
• Add functions ADDTIME and SUBTIME. (#132 #1194 #1252)
• Add Day_Of_Week Function As An Alias Of DayOfWeek (#190 #1228 #1239)
• Add Minute_Of_Hour Function As An Alias Of Minute Function (#1253)
• Add support for long value return for CEIL, CEILING and FLOOR math functions (#1205)
• Add Alternate Syntax For Match_Query And Other Functions (#1166)
• Add Support For TIME Type in DAY_OF_YEAR Functions (#199 #1223 1#258)
• Add Day_Of_Month Function As An Alias Of DayOfMonth (#1227 #1265)
• Add security patch for CVE-2020-15250 (#1095)
• Add security patch for CVE-2022-45868 (#1107)
• Add Hour_Of_Day Function As An Alias Of Hour (#1226 #1270)

Bug Fixes

• Fix FLOAT -> DOUBLE cast. (#1025)
• Fix error messaging from prometheus. (#1029 #1037)
• Add query function as alternate syntax to query_string function (#1010)
• Deprecate span collector (#990)
• back quote fix (#1041 #1050)
• Update DATE and TIME functions to parse string input as datetime (#991)
• Integ tests fix for arm64 (#1069)
• Fix history file usage in SQL CLI tool. (#1077)
• Update Jackson to 2.14.1 and fix dependency resolution issues (#1150)
• Change LIKE operator case-insensitive match (#1160)
• Fix arithmetic operator precedence (#1172 #1188)
• Fix back quoted alias of FROM subquery (#1189 #1208)
• Fix truncate function (#1197 #1213)
• Allow common keywords and scalar function name used as identifier (#1191 #1212)
• Suppress report uploading failure in CI. (#1180 #1220)
• Fixed error with single timestamp query (#1244 #1246 #1249)
• Support JOIN query on object field with unexpanded name (#1229 #1250)
• Bug fix for less than and greater than operators on @timestamp (#1267 #1272)

Infrastructure

• Change ApplicationContext lifecycle to per node level (#822)
• Catalog to Datasource changes (#1027 #1049 #1086)
• Bump jackson to 2.14.0 (#1058)
• Add metadatalog interface and default in memory implementation (#974)
• Decouple function repository and DSL from IoC container for use anywhere (#1085)
• Move DataSourceServiceImpl to core module (#1084)
• Improve pushdown optimizatio...

Version 2.4.0

Compatible with OpenSearch and OpenSearch Dashboards Version 2.4.0

Features

Data Source Management

• Catalog Implementation (#819)
• Catalog to Datasource changes (#1027)

Prometheus Support

• Prometheus Connector Initial Code (#878)
• Restricted catalog name to [a-zA-Z0-9_-] characters (#876)
• Table function for supporting prometheus query_range function (#875)
• List tables/metrics using information_schema in source command. (#914)
• [Backport 2.4] Prometheus select metric and stats queries. (#1020)

Log Pattern Command

• Add patterns and grok command (#813)

ML Command

• Add category_field to AD command in PPL (#952)
• A Generic ML Command in PPL (#971)

Enhancements

• Add datetime functions FROM_UNIXTIME and UNIX_TIMESTAMP (#835)
• Adding CONVERT_TZ and DATETIME functions to SQL and PPL (#848)
• Add Support for Highlight Wildcard in SQL (#827)
• Update SQL CLI to use AWS session token. (#918)
• Add typeof function. (#867)
• Show catalogs (#925)
• Add functions PERIOD_ADD and PERIOD_DIFF. (#933)
• Add take() aggregation function in PPL (#949)
• Describe Table with catalog name. (#989)
• Catalog Enhancements (#988)
• Rework on error reporting to make it more verbose and human-friendly. (#839)

Bug Fixes

• Fix EqualsAndHashCode Annotation Warning Messages (#847)
• Remove duplicated png file (#865)
• Fix NPE with multiple queries containing DOT(.) in index name. (#870)
• Update JDBC driver version (#941)
• Fix result order of parse with other run time fields (#934)
• AD timefield name issue (#919)
• [Backport 2.4] Add function name as identifier in antlr (#1018)
• [Backport 2.4] Fix incorrect results returned by min, max and avg (#1022)

Infrastructure

• Fix failing ODBC workflow (#828)
• Reorganize GitHub workflows. (#837)
• Update com.fasterxml.jackson to 2.13.4 to match opensearch repo. (#858)
• Trigger build on pull request synchronize action. (#873)
• Update Jetty Dependency (#872)
• Fix manual CI workflow and add name option. (#904)
• add groupId to pluginzip publication (#906)
• Enable ci for windows and macos (#907)
• Update group to groupId (#908)
• Enable ignored and disabled tests (#926)
• Update version of jackson-databind for sql-jdbc only (#943)
• Add security policy for ml-commons library (#945)
• Change condition to always upload coverage for linux workbench (#967)
• Bump ansi-regex for workbench (#975)
• Removed json-smart in the JDBC driver (#978)
• Update MacOS Version for ODBC Driver (#987)
• Update Jackson Databind version to 2.13.4.2 (#992)
• [Backport 2.4] Bump sql-cli version to 1.1.0 (#1024)

Documentation

• Add Forum link in SQL plugin README.md (#809)
• Fix indentation of patterns example (#880)
• Update docs - missing changes for #754. (#884)
• Fix broken links (#911)
• Adding docs related to catalog. (#963)
• SHOW CATALOGS documentation and integ tests (#977)
• [Backport 2.4] Add document for ml command. (#1017)

JDBC driver release 2.0.0.0

This version includes

• Bump JDBC driver's dependency version #735
• Update Jetty Dependency #872
• Update JDBC driver version #941
• Update version of jackson-databind for sql-jdbc only #943
• Removed json-smart in the JDBC driver #978

Thanks

@forestmvey, @GumpacG and @dai-chen for their valuable contribution.

Binaries

Please, see unsigned binaries below:

• shadow-copy-jar.zip
• no-shadow-copy-jar.zip

Notes

The signed driver should be published on Maven repository soon.

Version 2.3.0

Compatible with OpenSearch and OpenSearch Dashboards Version 2.3.0

Features

• Add maketime and makedate datetime functions (#755)

Enhancements

• Refactor implementation of relevance queries (#746)
• Extend query size limit using scroll (#716)
• Add any case of arguments in relevancy based functions to be allowed (#744)

Bug Fixes

• Fix unit test in PowerBI connector (#800)

Infrastructure

• Schedule request in worker thread (#748)
• Deprecated ClusterService and Using NodeClient to fetch metadata (#774)
• Change master node timeout to new API (#793)

Documentation

• Adding documentation about double quote implementation (#723)
• Add PPL security setting documentation (#777)
• Update PPL docs link for workbench (#758)

Version 2.2.0

Compatible with OpenSearch and OpenSearch Dashboards Version 2.2.0

Features

• Add match_phrase_prefix (#661)
• query_string Relevance Function Implementation in SQL and PPL (#675)
• Add Highlight In SQL (#717)

Enhancements

• implement transport api for PPL inter-plugin communication (#533)
• Two single or double quote escapes single or double quote when string is surrounded by same type of quote (#696)

Bug Fixes

• Reverted UseSSL flag to false and removed invalid test case (#671)
• Update BI connectors and drivers readme files (#665)
• Bump moment from 2.29.2 to 2.29.4 in /workbench (#702)

Infrastructure

• Staging for version increment automation (#684)
• Update tests and test data for relevancy search functions (#707)
• Remove ODFE BWC tests (#721)
• Github Actions fix for reference to OpenSearch-Dashboard not existing outside of main OS sql project (#704)

Documentation

• Update support link for Tableau connector (#643)
• Fix broken forum link (#694)
• Fix links in the doc file. (#705)

Maintenance

• change version bump under maintenance (#679)
• Bump version to 2.2.0 (#729)

Version 2.1.0.0

Version 2.1.0.0 Release Notes

Compatible with OpenSearch and OpenSearch Dashboards Version 2.1.0

Features

• Support match_phrase filter function in SQL and PPL (#604)
• Add implementation for simple_query_string relevance search function in SQL and PPL (#635)
• Add multi_match to SQL plugin (#649)
• Integ match bool prefix #187 (#634)
• PPL describe command (#646)

Maintenance

• change plugin folder name to opensearch-sql-plugin (#670)
• Version bump to 2.1.0 and gradle version bump (#655)

Bug Fixes

• Integ replace junit assertthat with hamcrest import (#616)
• Integ relevance function it fix (#608)
• Fix merge conflict on function name (#664)
• Fix fuzziness parsing in multi_match function. Update tests. (#668)
• ODBC SSL Compliance Fix (#653)

Infrastructure

• Match Query Unit Tests (#614)
• Uses custom plugin to publish zips to maven (#638)