Skip profile queries

[quangdutran]

Description

Profile search requests will be skipped during the search query construction in both request end and fail

Issues Resolved

[FEATURE] Filter out profile queries from the Top N queries

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dzane17]

@quangdutran Thanks for your contribution! I added some comments on how to fix the failing unit tests.

For future reference, you can run ./gradlew :test --tests "org.opensearch.*" (unit tests) and ./gradlew check (all repo checks) locally before submitting the PR to verify.

Also, there was a bug last week affecting OpenSearch plugins. You'll need to rebase your branch to include #296 before running the commands above.

[quangdutran]

Hi @dzane17
Thank you so much for the guide. I've followed your suggestion and re-run the gradle tests and check locally for the latest commit. Please help me to review it once you have time.

Cheers

[dzane17]

Testing

1. Start Query Insights ./gradlew run & index some data
2. Verify no top queries

% curl -XGET "http://localhost:9200/_insights/top_queries?pretty"                                                                        
{
  "top_queries" : [ ]
}

3. Search with profile=true

% curl -X GET "localhost:9200/my_index/_search" -H 'Content-Type: application/json' -d'             
{                        
  "profile": true,                                                                                                                                                 
  "query": {             
    "match": {                                                                                                                                                 
      "author": "Jane Smith"
    }
  }
}
'

{"took":138,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":2,"relation":"eq"},"max_score":0.31506687,"hits":[{"_index":"my_index","_id":"1","_score":0.31506687,"_source":{ "title": "OpenSearch Introduction", "content": "OpenSearch is a distributed search and analytics engine.", "author": "Jane Doe", "published_date": "2025-01-01" }},{"_index":"my_index","_id":"2","_score":0.31506687,"_source":{ "title": "Advanced OpenSearch Usage", "content": "This post dives deeper into OpenSearch features.", "author": "John Smith", "published_date": "2025-02-01" }}]},"profile":{"shards":[{"id":"[IHmlUUs5TxiLiE7Fg8w16Q][my_index][0]","inbound_network_time_in_millis":0,"outbound_network_time_in_millis":0,"searches":[{"query":[{"type":"BooleanQuery","description":"author:jane author:smith","time_in_nanos":38224186,"breakdown":{"set_min_competitive_score_count":0,"match_count":0,"shallow_advance_count":0,"next_doc":86466,"score_count":2,"compute_max_score_count":0,"advance":0,"advance_count":0,"score":24151,"shallow_advance":0,"create_weight_count":1,"build_scorer":34271047,"set_min_competitive_score":0,"match":0,"next_doc_count":3,"compute_max_score":0,"build_scorer_count":2,"create_weight":3842522},"children":[{"type":"TermQuery","description":"author:jane","time_in_nanos":4936522,"breakdown":{"set_min_competitive_score_count":0,"match_count":0,"shallow_advance_count":0,"next_doc":5182,"score_count":1,"compute_max_score_count":0,"advance":0,"advance_count":0,"score":9516,"shallow_advance":0,"create_weight_count":1,"build_scorer":2393326,"set_min_competitive_score":0,"match":0,"next_doc_count":2,"compute_max_score":0,"build_scorer_count":3,"create_weight":2528498}},{"type":"TermQuery","description":"author:smith","time_in_nanos":234097,"breakdown":{"set_min_competitive_score_count":0,"match_count":0,"shallow_advance_count":0,"next_doc":1594,"score_count":1,"compute_max_score_count":0,"advance":0,"advance_count":0,"score":3057,"shallow_advance":0,"create_weight_count":1,"build_scorer":56239,"set_min_competitive_score":0,"match":0,"next_doc_count":2,"compute_max_score":0,"build_scorer_count":3,"create_weight":173207}}]}],"rewrite_time":808871,"collector":[{"name":"TopScoreDocCollector","reason":"search_top_hits","time_in_nanos":424487}]}],"aggregations":[]}]}}

4. Verify no top queries

% curl -XGET "http://localhost:9200/_insights/top_queries?pretty"                                                                        
{
  "top_queries" : [ ]
}

5. Search with profile=false

% curl -X GET "localhost:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "profile": false,
  "query": {
    "match": {
      "author": "Jane Smith"
    }
  }
}
'

7. Verify top query present

% curl -XGET "http://localhost:9200/_insights/top_queries?pretty"                    
{
  "top_queries" : [
    {
      "timestamp" : 1744737154916,
      "id" : "16482b3c-2024-4042-9e82-a993cc97df8e",
      "group_by" : "NONE",
      "node_id" : "IHmlUUs5TxiLiE7Fg8w16Q",
      "source" : {
        "query" : {
          "match" : {
            "author" : {
              "query" : "Jane Smith",
              "operator" : "OR",
              "prefix_length" : 0,
              "max_expansions" : 50,
              "fuzzy_transpositions" : true,
              "lenient" : false,
              "zero_terms_query" : "NONE",
              "auto_generate_synonyms_phrase_query" : true,
              "boost" : 1.0
            }
          }
        }
      },
      "task_resource_usages" : [
        {
          "action" : "indices:data/read/search[phase/query]",
          "taskId" : 70,
          "parentTaskId" : 69,
          "nodeId" : "IHmlUUs5TxiLiE7Fg8w16Q",
          "taskResourceUsage" : {
            "cpu_time_in_nanos" : 5509000,
            "memory_in_bytes" : 224960
          }
        },
        {
          "action" : "indices:data/read/search",
          "taskId" : 69,
          "parentTaskId" : -1,
          "nodeId" : "IHmlUUs5TxiLiE7Fg8w16Q",
          "taskResourceUsage" : {
            "cpu_time_in_nanos" : 362000,
            "memory_in_bytes" : 3824
          }
        }
      ],
      "labels" : { },
      "total_shards" : 1,
      "phase_latency_map" : {
        "expand" : 0,
        "query" : 8,
        "fetch" : 0
      },
      "search_type" : "query_then_fetch",
      "indices" : [ ],
      "measurements" : {
        "cpu" : {
          "number" : 5871000,
          "count" : 1,
          "aggregationType" : "NONE"
        },
        "memory" : {
          "number" : 228784,
          "count" : 1,
          "aggregationType" : "NONE"
        },
        "latency" : {
          "number" : 11,
          "count" : 1,
          "aggregationType" : "NONE"
        }
      }
    }
  ]
}

[ansjcy]

@quangdutran thanks for the contribution! the PR looks good to me as well, merging it!