Move query categorization changes to query insights plugin

[deshsidd]

Query categorization changes to increment counters for search query related metrics currently resides on the search path and occurs before the request.

Move these changes to the query insights plugin and make sure the incrementing of counters happens separately from the search path.

Another option is to keep query categorization changes as is. However, this will lead to additional overhead on the search path. Furthermore, we need to tie query latency, cpu, memory with the query categorization data which will only be possible if we increment the counters after the request is completed and the query latency and resource usage data resides inside the plugin.

To support the above and to prevent doing these counter increments on the search path, we need to move query categorization changes to the query insights plugin.

Related Issues

Resolves #14527
Addresses #11596

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 9e5a09a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for af58c5f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 2bd1a05: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 68a226e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 8dc38c4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for a4ad58e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for cdce692: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for dc1fefb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 0906513: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for be6527b: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 67.69231% with 21 lines in your changes missing coverage. Please review.

Project coverage is 71.78%. Comparing base (a34270d) to head (eb8ebd8).
Report is 5 commits behind head on main.

Files	Patch %	Lines
...in/insights/core/service/QueryInsightsService.java	46.66%	7 Missing and 1 partial ⚠️
...nsearch/plugin/insights/rules/model/Attribute.java	73.91%	3 Missing and 3 partials ⚠️
...re/service/categorizer/SearchQueryCategorizer.java	37.50%	5 Missing ⚠️
...categorizer/SearchQueryAggregationCategorizer.java	50.00%	1 Missing ⚠️
...insights/settings/QueryCategorizationSettings.java	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##               main   #14528    +/-   ##
==========================================
  Coverage     71.77%   71.78%            
+ Complexity    62302    62282    -20     
==========================================
  Files          5125     5127     +2     
  Lines        292473   292547    +74     
  Branches      42258    42269    +11     
==========================================
+ Hits         209912   209991    +79     
+ Misses        65324    65213   -111     
- Partials      17237    17343   +106     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

❕ Gradle check result for eb8ebd8: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

❌ Gradle check result for d419ca5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for d6b29c7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[jainankitk]

Why do you need a volatile boolean? Does the eventual consistency not work here?

[deshsidd]

If searchQueryMetricsEnabled is not declared as volatile, changes made by one thread to searchQueryMetricsEnabled may not be immediately visible to other threads and could cause some issues.

[jainankitk]

nit: Should we really have service in the namespace?

[deshsidd]

Had added categorizer in core earlier but @ansjcy mentioned that it might make more sense to keep it under service. I am okay either way.

[jainankitk]

Most of the QueryCategorizer logic is independent of QueryInsightsService. Hence, categorizer should not be contained within service IMO.

[github-actions]

❌ Gradle check result for 9b80e90: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[jainankitk]

@deshsidd - This PR can be discarded in favor of opensearch-project/query-insights#16, right?

[deshsidd]

Closing in favor of : https://github.com/opensearch-project/OpenSearch/pull/14528/files