-
Notifications
You must be signed in to change notification settings - Fork 138
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Query rewrite with secondary index and materialized view #1435
Comments
Nice. Will this be capable of acceleration (across fields) in case of covering index with selective fields? Say, if we index only field 'a' into opensearch and the query is something like 'avg(b) where a startswith "blah"'. OpenSearch will answer the prefix query on 'blah*' since it is indexed, but are we planing to use those results in the spark scan for b while aggregating? |
@muralikpbhat Thanks for the comment! I think your example is more for fine-grained filtering index. Actually we're focused on the following index data structure:
|
I'm closing this because query rewrite for MV is deprioritized. The main reason is performance overhead introduced by current logical integration between OpenSearch and Spark. Thinking of MV as a managed Maximus table and rewrite query in Spark make sense. However, the cost of transferring data between Spark and OpenSearch maybe high. Meanwhile there are limitations in Spark SQL support for OpenSearch Dashboard which requires extension. So we'd like to make MV data a regular OpenSearch index so OpenSearch can access it directly in current initial phase. |
Is your feature request related to a problem?
In #1379 and #1407, we're able to build secondary index and materialized view on Maximus table. However, there are still some work remaining for both query acceleration:
What solution would you like?
Rewrite query from user with available secondary index or materialized view on the Maximus table to to accelerate its execution.
What alternatives have you considered?
Consider query rewrite limitation and out of scope for now.
Do you have any additional context?
N/A
The text was updated successfully, but these errors were encountered: