-
Notifications
You must be signed in to change notification settings - Fork 139
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for pagination in v2 engine of SELECT * FROM <table>
queries
#1666
Conversation
* Support pagination in V2 engine, phase 1 (#226) * Fixing integration tests broken during POC Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Comment to clarify an exception. Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Add support for paginated scroll request, first page. Implement PaginatedPlanCache.convertToPlan for second page to work. Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Progress on paginated scroll request, subsequent page. Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Move `ExpressionSerializer` from `opensearch` to `core`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Rename `Cursor` `asString` to `toString`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Disable scroll cleaning. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Add full cursor serialization and deserialization. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Misc fixes. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Further work on pagination. * Added push down page size from `LogicalPaginate` to `LogicalRelation`. * Improved cursor encoding and decoding. * Added cursor compression. * Fixed issuing `SearchScrollRequest`. * Fixed returning last empty page. * Minor code grooming/commenting. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Pagination fix for empty indices. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix error reporting on wrong cursor. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Minor comments and error reporting improvement. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Add an end-to-end integration test. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Add `explain` request handlers. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Add IT for explain. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Address issues flagged by checkstyle build step (#229) Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Pagination, phase 1: Add unit tests for `:core` module with coverage. (#230) * Add unit tests for `:core` module with coverage. Uncovered: `toCursor`, because it is will be changed soon. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Pagination, phase 1: Add unit tests for SQL module with coverage. (#239) * Add unit tests for SQL module with coverage. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Update sql/src/main/java/org/opensearch/sql/sql/domain/SQLQueryRequest.java Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> --------- Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> * Pagination, phase 1: Add unit tests for `:opensearch` module with coverage. (#233) * Add UT for `:opensearch` module with full coverage, except `toCursor`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix checkstyle. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> --------- Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix the merges. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix explain. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix scroll cleaning. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Store `TotalHits` and use it to report `total` in response. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Add missing UT for `:protocol` module. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix PPL UTs damaged in f4ea4ad. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Minor checkstyle fixes. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fallback to v1 engine for pagination (#245) * Pagination fallback integration tests. Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Add UT with coverage for `toCursor` serialization. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix broken tests in `legacy`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix getting `total` from non-paged requests and from queries without `FROM` clause. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix scroll cleaning. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix cursor request processing. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Update ITs. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix (again) TotalHits feature. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix typo in prometheus config. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Recover commented logging. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Move `test_pagination_blackbox` to a separate class and add logging. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Address some PR feedbacks: rename some classes and revert unnecessary whitespace changed. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Minor commenting. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Address PR comments. * Add javadocs * Renames * Cleaning up some comments * Remove unused code * Speed up IT Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Minor missing changes. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Integration tests for fetch_size, max_result_window, and query.size_limit (#248) Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Remove `PaginatedQueryService`, extend `QueryService` to hold two planners and use them. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Move push down functions from request builders to a new interface. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Some file moves. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Minor clean-up according to PR review. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: MaxKsyunz <maxk@bitquilltech.com> Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> Co-authored-by: Max Ksyunz <max.ksyunz@improving.com> * Make scroll timeout configurable. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Fix IT to set cursor keep alive parameter. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Remove `QueryId.None`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Rename according to PR feedback. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Remove default implementations of `PushDownRequestBuilder`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Merge paginated plan optimizer into the regular optimizer. (#1516) Merge paginated plan optimizer into the regular optimizer. --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Complete rework on serialization and deserialization. (#1498) Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Resolve merge conflicts and fix tests. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Minor cleanup. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Minor cleanup - missing changes for the previous commit. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Remove paginate operator (#1528) * Remove PaginateOperator class since it is no longer used. --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> * Remove `PaginatedPlan` - move logic to `QueryPlan`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Remove default implementations from `SerializablePlan`. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Add a doc. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Update design graphs. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * More fixes for merge from upstream/main. Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: MaxKsyunz <maxk@bitquilltech.com> Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> Co-authored-by: Max Ksyunz <max.ksyunz@improving.com>
…integ Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
* Add newer docs for pagination. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Address PR feedback. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Complete TODO and add some more info. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Address doc review comments. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Clean up docs. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Apply suggestions from code review Co-authored-by: Andrew Carbonetto <andrewc@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Apply suggestions from code review Co-authored-by: Andrew Carbonetto <andrewc@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> * Minor fixes. Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> --------- Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: Andrew Carbonetto <andrewc@bitquilltech.com>
…integ Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
* Update design document to reflect refactor. Signed-off-by: Max Ksyunz <maxk@bitquilltech.com> Co-authored-by: Andrew Carbonetto <andrewc@bitquilltech.com>
Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>
Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Signed-off-by: Max Ksyunz <maxk@bitquilltech.com> Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Codecov Report
@@ Coverage Diff @@
## main #1666 +/- ##
============================================
+ Coverage 97.18% 97.24% +0.06%
- Complexity 4150 4259 +109
============================================
Files 372 386 +14
Lines 10429 10668 +239
Branches 716 738 +22
============================================
+ Hits 10135 10374 +239
Misses 287 287
Partials 7 7
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
* Remove `getTotalHits` feature. --------- Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
… feature/pagination/integ
Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>
* Convert a scroll request to string that can be included in a cursor. | ||
* @return a string representing the scroll request. | ||
*/ | ||
@Override |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- please correct the doc, it is not alligned with code.
- add the doc in OpenSearchRequest.
@@ -24,36 +24,39 @@ | |||
* by delegated builder internally. This is to avoid conditional check of different push down logic | |||
* for non-aggregate and aggregate query everywhere. | |||
*/ | |||
public class OpenSearchIndexScanBuilder extends TableScanBuilder { | |||
public abstract class OpenSearchIndexScanBuilder extends TableScanBuilder { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
follow up on same question on #1600.
why it is abstract and no subclass?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The subclass is created in OpenSearchIndex.createScanBuilder
. The subclass captures data (primarily OpenSearch client reference) necessary in OpenSearchIndexScanBuilder.build
to create an instance of OpenSearchIndexScan
.
At this point, OpenSearchIndexScanBuilder
can become an inner class of OpenSearchIndex
but doing so includes refactoring of OpenSearchIndexScanOptimizationTest suite as well.
I plan to include this change as part of follow-up pagination work.
This was my response in #1600 with a slightly different description.
} | ||
|
||
@Override | ||
public TableScanOperator build() { | ||
return delegate.build(); | ||
return createScan(delegate.build()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what is the difference of build and createScan?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OpenSearchIndexScanBuilder.createScan
-- create an OpenSearchIndexScan based on the provided OpenSearchRequest.
OpenSearchIndexScanBuilder.build
-- build an OpenSearchRequest and an OpenSearchIndexScan based on it.
They will be merged as part of making OpenSearchIndexScanBuilder
inner class of OpenSearchIndexScan
.
Related to the other discussion about builder.
Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>
Signed-off-by: MaxKsyunz <maxk@bitquilltech.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Observation: there are about 15 files with no-op changes - whitespaces or comment updating. Pagination feature had bigger scope, but after few rounds of reworks and improvements, some changes were reverted, but not comments.
I also found few typos in javadocs, nothing critical. Will be fixed in the following work on Pagination feature.
No issues nor objections found while code review and manual testing!
#1666) v2 SQL engine can now paginate simple queries. Pagination is initiated by setting fetch_size property in the request JSON. Pagination is implemented using the OpenSearch Scroll API. Please see pagination-v2.md for implementation details. --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Signed-off-by: Max Ksyunz <maxk@bitquilltech.com> Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> Co-authored-by: Andrew Carbonetto <andrewc@bitquilltech.com> (cherry picked from commit 57ce303)
#1666) v2 SQL engine can now paginate simple queries. Pagination is initiated by setting fetch_size property in the request JSON. Pagination is implemented using the OpenSearch Scroll API. Please see pagination-v2.md for implementation details. --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Signed-off-by: Max Ksyunz <maxk@bitquilltech.com> Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> Co-authored-by: Andrew Carbonetto <andrewc@bitquilltech.com> (cherry picked from commit 57ce303)
…<table>` queries (#1685) * Support for pagination in v2 engine of `SELECT * FROM <table>` queries (#1666) v2 SQL engine can now paginate simple queries. Pagination is initiated by setting fetch_size property in the request JSON. Pagination is implemented using the OpenSearch Scroll API. Please see pagination-v2.md for implementation details. --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Signed-off-by: Max Ksyunz <maxk@bitquilltech.com> Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> Co-authored-by: Andrew Carbonetto <andrewc@bitquilltech.com> (cherry picked from commit 57ce303)
…<table>` queries (#1684) * Support for pagination in v2 engine of `SELECT * FROM <table>` queries (#1666) v2 SQL engine can now paginate simple queries. Pagination is initiated by setting fetch_size property in the request JSON. Pagination is implemented using the OpenSearch Scroll API. Please see pagination-v2.md for implementation details. --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Signed-off-by: Max Ksyunz <maxk@bitquilltech.com> Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> Co-authored-by: Andrew Carbonetto <andrewc@bitquilltech.com> (cherry picked from commit 57ce303) * Fix test build failure. Somehow the import is required in 2.x but not 3.0 Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Co-authored-by: Max Ksyunz <maxk@bitquilltech.com>
opensearch-project#1666) v2 SQL engine can now paginate simple queries. Pagination is initiated by setting fetch_size property in the request JSON. Pagination is implemented using the OpenSearch Scroll API. Please see pagination-v2.md for implementation details. --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Signed-off-by: Max Ksyunz <maxk@bitquilltech.com> Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> Co-authored-by: Andrew Carbonetto <andrewc@bitquilltech.com>
opensearch-project#1666) v2 SQL engine can now paginate simple queries. Pagination is initiated by setting fetch_size property in the request JSON. Pagination is implemented using the OpenSearch Scroll API. Please see pagination-v2.md for implementation details. --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Signed-off-by: Max Ksyunz <maxk@bitquilltech.com> Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> Co-authored-by: Andrew Carbonetto <andrewc@bitquilltech.com> Signed-off-by: Mitchell Gale <Mitchell.Gale@improving.com>
opensearch-project#1666) v2 SQL engine can now paginate simple queries. Pagination is initiated by setting fetch_size property in the request JSON. Pagination is implemented using the OpenSearch Scroll API. Please see pagination-v2.md for implementation details. --------- Signed-off-by: MaxKsyunz <maxk@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com> Signed-off-by: Max Ksyunz <maxk@bitquilltech.com> Co-authored-by: Yury-Fridlyand <yury.fridlyand@improving.com> Co-authored-by: GabeFernandez310 <Gabriel.Fernandez@improving.com> Co-authored-by: Andrew Carbonetto <andrewc@bitquilltech.com> Signed-off-by: Mitchell Gale <Mitchell.Gale@improving.com>
Description
v2 SQL engine can now paginate simple queries. Pagination is initiated by setting
fetch_size
property in the request JSON.For example, initiate a pagination request, send:
Send the following to get a subsequent page:
Each response to a pagination request will include a
cursor
property if there is more data available. The last page in the paginated request will not have acursor
property.Pagination is implemented using the OpenSearch Scroll API. Please see pagination-v2.md for implementation details.
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.