Add setting to exclude certain indices from insight query #308

quangdutran · 2025-04-17T06:09:55Z

Signed-off-by: Du Tran quangdutran809@gmail.com

Description

Add setting excluded_indices to let user exclude certain indices from insight query

Issues Resolved

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

src/main/java/org/opensearch/plugin/insights/core/listener/QueryInsightsListener.java

src/main/java/org/opensearch/plugin/insights/settings/QueryInsightsSettings.java

src/main/java/org/opensearch/plugin/insights/core/listener/QueryInsightsListener.java

src/main/java/org/opensearch/plugin/insights/core/service/QueryInsightsService.java

src/main/java/org/opensearch/plugin/insights/core/listener/QueryInsightsListener.java

src/main/java/org/opensearch/plugin/insights/core/service/QueryInsightsService.java

src/test/java/org/opensearch/plugin/insights/core/service/QueryInsightsServiceTests.java

src/test/java/org/opensearch/plugin/insights/core/listener/QueryInsightsListenerTests.java

src/main/java/org/opensearch/plugin/insights/core/listener/QueryInsightsListener.java

dzane17 · 2025-04-23T16:51:28Z

src/main/java/org/opensearch/plugin/insights/core/listener/QueryInsightsListener.java

    private boolean groupingFieldNameEnabled;
    private boolean groupingFieldTypeEnabled;
    private final QueryShapeGenerator queryShapeGenerator;
+    private Set<String> excludedIndicesHashSet;


We should precompute the excluded index regex patterns. Can you change excludedIndicesHashSet from Set<String> to Set<Pattern>? Will need to update the logic where excludedIndicesHashSet is used:

for (Pattern indexPattern : excludedIndexPatterns) { if (indexPattern.matcher(indexName).matches()) { return true; } }

Precomputation would be inside the setter right? and the purpose of this is to create the regex pattern beforehands, so this won't be executed for every request, is it

And one more thing is that if we convert to Pattern then I have to adding dot in front of the wildcard * to make it work or is there any other way to do it

Let's go with the Pattern approach and substitute the wildcards.

I did a quick benchmark on Regex.simpleMatch vs precompiled pattern .matcher(testString).matches(). The Regex method has a high initial cost, then benefits from some caching. However the Pattern option is still more efficient on average.

1> [2568-04-25T00:39:55,920][INFO ][o.o.p.i.c.r.MultiIndexDateRangeIT] [testPerf] Timing results for Regex.simpleMatch: 1> [2568-04-25T00:39:55,923][INFO ][o.o.p.i.c.r.MultiIndexDateRangeIT] [testPerf] Run 1: 2.062202 ms 1> [2568-04-25T00:39:55,923][INFO ][o.o.p.i.c.r.MultiIndexDateRangeIT] [testPerf] Run 2: 0.002459 ms 1> [2568-04-25T00:39:55,923][INFO ][o.o.p.i.c.r.MultiIndexDateRangeIT] [testPerf] Run 3: 0.003216 ms 1> [2568-04-25T00:39:55,924][INFO ][o.o.p.i.c.r.MultiIndexDateRangeIT] [testPerf] Run 4: 0.002384 ms 1> [2568-04-25T00:39:55,924][INFO ][o.o.p.i.c.r.MultiIndexDateRangeIT] [testPerf] Run 5: 0.003177 ms 1> [2568-04-25T00:39:55,924][INFO ][o.o.p.i.c.r.MultiIndexDateRangeIT] [testPerf] Run 6: 0.002605 ms 1> [2568-04-25T00:39:55,924][INFO ][o.o.p.i.c.r.MultiIndexDateRangeIT] [testPerf] Run 7: 0.002438 ms 1> [2568-04-25T00:39:55,925][INFO ][o.o.p.i.c.r.MultiIndexDateRangeIT] [testPerf] Run 8: 0.002981 ms 1> [2568-04-25T00:39:55,925][INFO ][o.o.p.i.c.r.MultiIndexDateRangeIT] [testPerf] Run 9: 0.002324 ms 1> [2568-04-25T00:39:55,925][INFO ][o.o.p.i.c.r.MultiIndexDateRangeIT] [testPerf] Run 10: 0.003247 ms 1> [2568-04-25T00:39:55,926][INFO ][o.o.p.i.c.r.MultiIndexDateRangeIT] [testPerf] Timing results for Precompiled Pattern match: 1> [2568-04-25T00:39:55,926][INFO ][o.o.p.i.c.r.MultiIndexDateRangeIT] [testPerf] Run 1: 0.013073 ms 1> [2568-04-25T00:39:55,926][INFO ][o.o.p.i.c.r.MultiIndexDateRangeIT] [testPerf] Run 2: 0.00164 ms 1> [2568-04-25T00:39:55,927][INFO ][o.o.p.i.c.r.MultiIndexDateRangeIT] [testPerf] Run 3: 0.002374 ms 1> [2568-04-25T00:39:55,927][INFO ][o.o.p.i.c.r.MultiIndexDateRangeIT] [testPerf] Run 4: 0.00258 ms 1> [2568-04-25T00:39:55,927][INFO ][o.o.p.i.c.r.MultiIndexDateRangeIT] [testPerf] Run 5: 0.001796 ms 1> [2568-04-25T00:39:55,927][INFO ][o.o.p.i.c.r.MultiIndexDateRangeIT] [testPerf] Run 6: 0.001598 ms 1> [2568-04-25T00:39:55,928][INFO ][o.o.p.i.c.r.MultiIndexDateRangeIT] [testPerf] Run 7: 0.001615 ms 1> [2568-04-25T00:39:55,928][INFO ][o.o.p.i.c.r.MultiIndexDateRangeIT] [testPerf] Run 8: 0.001556 ms 1> [2568-04-25T00:39:55,928][INFO ][o.o.p.i.c.r.MultiIndexDateRangeIT] [testPerf] Run 9: 0.001392 ms 1> [2568-04-25T00:39:55,928][INFO ][o.o.p.i.c.r.MultiIndexDateRangeIT] [testPerf] Run 10: 0.001359 ms

src/main/java/org/opensearch/plugin/insights/core/listener/QueryInsightsListener.java

deshsidd · 2025-04-23T20:40:17Z

@quangdutran Build failing can you please take a look. Thanks!

dzane17 · 2025-04-25T17:56:26Z

Thanks @quangdutran. The code looks good to me. Could you lastly add testing steps & output in the PR description to demonstrate the feature.
Example: #298 (comment)

quangdutran · 2025-04-27T06:51:28Z

Testing

1. Start Query Insights ./gradlew run & create test data on three different indices named: products, products_archive, equipments
2. Set the excluded index named products
Request:

curl --location --request PUT 'http://localhost:9200/_cluster/settings' \
--header 'Content-Type: application/json' \
--data '{
  "persistent" : {
    "search.insights.top_queries.excluded_indices" : "products"
  }
}

'

Response:

{
    "acknowledged": true,
    "persistent": {
        "search": {
            "insights": {
                "top_queries": {
                    "excluded_indices": "products"
                }
            }
        }
    },
    "transient": {}
}

3. Perform search on products and check the top queries
Search:

$ curl -u admin:admin -XGET "http://localhost:9200/products/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": {
      "name": "keyboard"
    }
  }
}'

{"took" : 54,"timed_out" : false,"_shards" : {"total" : 1,"successful" : 1,"skipped" : 0,"failed" : 0},"hits" : {"total" : {"value" : 2,"relation" : "eq"},"max_score" : 1.708163,"hits" : [{"_index" : "products","_id" : "12","_score" : 1.708163,"_source" : {"name" : "Bluetooth Keyboard","category" : "Clothing","price" : 168.55,"stock" : 321,"created_at" : "2024-04-19T00:00:00Z"}},{"_index" : "products","_id" : "71","_score" : 1.708163,"_source" : {"name" : "Bluetooth Keyboard","category" : "Toys","price" : 79.46,"stock" : 107,"created_at" : "2024-04-08T00:00:00Z"}}]}}

Top queries:

curl --location 'http://localhost:9200/_insights/top_queries' 

{     "top_queries": [] }

Perform search on products_archive and equipments and check the top queries
Search:

$ curl -u admin:admin -XGET "http://localhost:9200/products_archive/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "range": {
      "price": {
        "gte": 50,
        "lte": 100
      }
    }
  }
}'

{"took" : 21,"timed_out" : false,"_shards" : {"total" : 1,"successful" : 1,"skipped" : 0,"failed" : 0},"hits" : {"total" : {"value" : 20,"relation" : "eq"},"max_score" : 1.0,"hits" : [{"_index" : "products_archive","_id" : "9","_score" : 1.0,"_source" : {"name" : "Noise Cancelling Headphones","category" : "Sports","price" : 62.7,"stock" : 109,"created_at" : "2024-04-01T00:00:00Z"}},{"_index" : "products_archive","_id" : "11","_score" : 1.0,"_source" : {"name" : "Fitness Tracker","category" : "Electronics","price" : 52.65,"stock" : 451,"created_at" : "2024-05-26T00:00:00Z"}},{"_index" : "products_archive","_id" : "13","_score" : 1.0,"_source" : {"name" : "Action Camera","category" : "Toys","price" : 68.62,"stock" : 103,"created_at" : "2024-04-19T00:00:00Z"}},{"_index" : "products_archive","_id" : "17","_score" : 1.0,"_source" : {"name" : "Action Camera","category" : "Electronics","price" : 58.74,"stock" : 47,"created_at" : "2024-05-12T00:00:00Z"}},{"_index" : "products_archive","_id" : "18","_score" : 1.0,"_source" : {"name" : "HDMI Cable","category" : "Toys","price" : 98.31,"stock" : 158,"created_at" : "2024-05-27T00:00:00Z"}},{"_index" : "products_archive","_id" : "19","_score" : 1.0,"_source" : {"name" : "E-book Reader","category" : "Accessories","price" : 57.75,"stock" : 235,"created_at" : "2024-05-04T00:00:00Z"}},{"_index" : "products_archive","_id" : "29","_score" : 1.0,"_source" : {"name" : "E-book Reader","category" : "Electronics","price" : 68.73,"stock" : 230,"created_at" : "2024-04-14T00:00:00Z"}},{"_index" : "products_archive","_id" : "31","_score" : 1.0,"_source" : {"name" : "Wireless Mouse","category" : "Clothing","price" : 74.91,"stock" : 463,"created_at" : "2024-05-03T00:00:00Z"}},{"_index" : "products_archive","_id" : "36","_score" : 1.0,"_source" : {"name" : "LED Desk Lamp","category" : "Home","price" : 59.49,"stock" : 214,"created_at" : "2024-05-04T00:00:00Z"}},{"_index" : "products_archive","_id" : "43","_score" : 1.0,"_source" : {"name" : "Noise Cancelling Headphones","category" : "Books","price" : 55.04,"stock" : 349,"created_at" : "2024-05-15T00:00:00Z"}}]}}


curl -u admin:admin -XGET "http://localhost:9200/equipments/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "must": [
        { "term": { "category": "Electronics" } },
        { "range": { "price": { "lt": 80 } } }
      ]
    }
  }
}'

{   "took" : 8,   "timed_out" : false,   "_shards" : {     "total" : 1,     "successful" : 1,     "skipped" : 0,     "failed" : 0   },   "hits" : {     "total" : {       "value" : 0,       "relation" : "eq"     },     "max_score" : null,     "hits" : [ ]   } }

Top queries:

curl --location 'http://localhost:9200/_insights/top_queries'

{"top_queries": [{"timestamp": 1745735725482,"id": "610e09ca-92eb-4bea-b377-09960cd59eb3","search_type": "query_then_fetch","group_by": "NONE","phase_latency_map": {"expand": 0,"query": 15,"fetch": 0},"source": {"query": {"range": {"price": {"from": 50,"to": 100,"include_lower": true,"include_upper": true,"boost": 1.0}}}},"task_resource_usages": [{"action": "indices:data/read/search[phase/query]","taskId": 269,"parentTaskId": 268,"nodeId": "wMTHS3XHRNijDg92_yyL3A","taskResourceUsage": {"cpu_time_in_nanos": 15625000,"memory_in_bytes": 1843904}},{"action": "indices:data/read/search","taskId": 268,"parentTaskId": -1,"nodeId": "wMTHS3XHRNijDg92_yyL3A","taskResourceUsage": {"cpu_time_in_nanos": 0,"memory_in_bytes": 3904}}],"indices": ["products_archive"],"node_id": "wMTHS3XHRNijDg92_yyL3A","total_shards": 1,"labels": {},"measurements": {"cpu": {"number": 15625000,"count": 1,"aggregationType": "NONE"},"latency": {"number": 19,"count": 1,"aggregationType": "NONE"},"memory": {"number": 1847808,"count": 1,"aggregationType": "NONE"}}},{"timestamp": 1745735809155,"id": "e8b47f13-65d0-4571-801d-7866acfa0dfe","search_type": "query_then_fetch","group_by": "NONE","phase_latency_map": {"expand": 0,"query": 7,"fetch": 0},"source": {"query": {"bool": {"must": [{"term": {"category": {"value": "Electronics","boost": 1.0}}},{"range": {"price": {"from": null,"to": 80,"include_lower": true,"include_upper": false,"boost": 1.0}}}],"adjust_pure_negative": true,"boost": 1.0}}},"task_resource_usages": [{"action": "indices:data/read/search[phase/query]","taskId": 297,"parentTaskId": 296,"nodeId": "wMTHS3XHRNijDg92_yyL3A","taskResourceUsage": {"cpu_time_in_nanos": 0,"memory_in_bytes": 485216}},{"action": "indices:data/read/search","taskId": 296,"parentTaskId": -1,"nodeId": "wMTHS3XHRNijDg92_yyL3A","taskResourceUsage": {"cpu_time_in_nanos": 0,"memory_in_bytes": 3792}}],"indices": ["equipments"],"node_id": "wMTHS3XHRNijDg92_yyL3A","total_shards": 1,"labels": {},"measurements": {"cpu": {"number": 0,"count": 1,"aggregationType": "NONE"},"latency": {"number": 8,"count": 1,"aggregationType": "NONE"},"memory": {"number": 489008,"count": 1,"aggregationType": "NONE"}}}]}

5. Set the excluded indices to products*

curl --location --request PUT 'http://localhost:9200/_cluster/settings' \
--header 'Content-Type: application/json' \
--data '{
  "persistent" : {
    "search.insights.top_queries.excluded_indices" : "products*"
  }
}
'

{
    "acknowledged": true,
    "persistent": {
        "search": {
            "insights": {
                "top_queries": {
                    "excluded_indices": "products*"
                }
            }
        }
    },
    "transient": {}
}

6. Perform search on index products and products_archive again and check the top queries
Search:

$ curl -u admin:admin -XGET "http://localhost:9200/products/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": {
      "name": "hub"
    }
  }
}'

{"took" : 2,"timed_out" : false,"_shards" : {"total" : 1,"successful" : 1,"skipped" : 0,"failed" : 0},"hits" : {"total" : {"value" : 6,"relation" : "eq"},"max_score" : 1.0559033,"hits" : [{"_index" : "products","_id" : "35","_score" : 1.0559033,"_source" : {"name" : "USB-C Hub","category" : "Home","price" : 185.87,"stock" : 341,"created_at" : "2024-04-30T00:00:00Z"}},{"_index" : "products","_id" : "50","_score" : 1.0559033,"_source" : {"name" : "USB-C Hub","category" : "Sports","price" : 73.02,"stock" : 313,"created_at" : "2024-05-19T00:00:00Z"}},{"_index" : "products","_id" : "54","_score" : 1.0559033,"_source" : {"name" : "USB-C Hub","category" : "Sports","price" : 279.39,"stock" : 437,"created_at" : "2024-05-08T00:00:00Z"}},{"_index" : "products","_id" : "58","_score" : 1.0559033,"_source" : {"name" : "USB-C Hub","category" : "Home","price" : 153.14,"stock" : 126,"created_at" : "2024-05-16T00:00:00Z"}},{"_index" : "products","_id" : "62","_score" : 1.0559033,"_source" : {"name" : "USB-C Hub","category" : "Accessories","price" : 95.66,"stock" : 300,"created_at" : "2024-04-30T00:00:00Z"}},{"_index" : "products","_id" : "75","_score" : 1.0559033,"_source" : {"name" : "USB-C Hub","category" : "Home","price" : 99.77,"stock" : 421,"created_at" : "2024-04-18T00:00:00Z"}}]}}

$ curl -u admin:admin -XGET "http://localhost:9200/products_archive/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": {
      "name": "shoes"
    }
  }
}'

{"took" : 1,"timed_out" : false,"_shards" : {"total" : 1,"successful" : 1,"skipped" : 0,"failed" : 0},"hits" : {"total" : {"value" : 3,"relation" : "eq"},"max_score" : 1.5527762,"hits" : [{"_index" : "products_archive","_id" : "1","_score" : 1.5527762,"_source" : {"name" : "Running Shoes","category" : "Clothing","price" : 287.26,"stock" : 172,"created_at" : "2024-04-12T00:00:00Z"}},{"_index" : "products_archive","_id" : "64","_score" : 1.5527762,"_source" : {"name" : "Running Shoes","category" : "Books","price" : 108.41,"stock" : 496,"created_at" : "2024-05-19T00:00:00Z"}},{"_index" : "products_archive","_id" : "95","_score" : 1.5527762,"_source" : {"name" : "Running Shoes","category" : "Accessories","price" : 171.87,"stock" : 50,"created_at" : "2024-05-19T00:00:00Z"}}]}}

Top queries:

curl --location 'http://localhost:9200/_insights/top_queries'

{
    "top_queries": []
}

7. Set the excluded indices to null to reset the setting

curl --location --request PUT 'http://localhost:9200/_cluster/settings' \
--header 'Content-Type: application/json' \
--data '{
  "persistent" : {
    "search.insights.top_queries.excluded_indices" : null
  }
}
'

{
    "acknowledged": true,
    "persistent": {},
    "transient": {}
}

8. Perform search on any index and check top queries

$ curl -u admin:admin -XGET "http://localhost:9200/products_archive/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": {
      "name": "shoes"
    }
  }
}'
{"took" : 2,"timed_out" : false,"_shards" : {"total" : 1,"successful" : 1,"skipped" : 0,"failed" : 0},"hits" : {"total" : {"value" : 3,"relation" : "eq"},"max_score" : 1.5527762,"hits" : [{"_index" : "products_archive","_id" : "1","_score" : 1.5527762,"_source" : {"name" : "Running Shoes","category" : "Clothing","price" : 287.26,"stock" : 172,"created_at" : "2024-04-12T00:00:00Z"}},{"_index" : "products_archive","_id" : "64","_score" : 1.5527762,"_source" : {"name" : "Running Shoes","category" : "Books","price" : 108.41,"stock" : 496,"created_at" : "2024-05-19T00:00:00Z"}},{"_index" : "products_archive","_id" : "95","_score" : 1.5527762,"_source" : {"name" : "Running Shoes","category" : "Accessories","price" : 171.87,"stock" : 50,"created_at" : "2024-05-19T00:00:00Z"}}]}}

curl --location 'http://localhost:9200/_insights/top_queries'
{"top_queries": [{"timestamp": 1745736586196,"id": "e6fb20fb-f01b-4a47-ac18-b1a574202665","search_type": "query_then_fetch","group_by": "NONE","phase_latency_map": {"expand": 0,"query": 1,"fetch": 0},"source": {"query": {"match": {"name": {"query": "shoes","operator": "OR","prefix_length": 0,"max_expansions": 50,"fuzzy_transpositions": true,"lenient": false,"zero_terms_query": "NONE","auto_generate_synonyms_phrase_query": true,"boost": 1.0}}}},"task_resource_usages": [{"action": "indices:data/read/search[phase/query]","taskId": 726,"parentTaskId": 725,"nodeId": "wMTHS3XHRNijDg92_yyL3A","taskResourceUsage": {"cpu_time_in_nanos": 0,"memory_in_bytes": 45144}},{"action": "indices:data/read/search","taskId": 725,"parentTaskId": -1,"nodeId": "wMTHS3XHRNijDg92_yyL3A","taskResourceUsage": {"cpu_time_in_nanos": 0,"memory_in_bytes": 3880}}],"indices": ["products_archive"],"node_id": "wMTHS3XHRNijDg92_yyL3A","total_shards": 1,"labels": {},"measurements": {"cpu": {"number": 0,"count": 1,"aggregationType": "NONE"},"latency": {"number": 1,"count": 1,"aggregationType": "NONE"},"memory": {"number": 49024,"count": 1,"aggregationType": "NONE"}}}]}

ansjcy

The overall logic looks pretty good to me, @quangdutran thanks for the change!!

Left several comments. Also, could you add an integration test for this change? - We don't have integ tests for our settings endpoints, but I believe this is an important feature we should have integ test coverage. You can look at TopQueriesRestIT for reference.

src/main/java/org/opensearch/plugin/insights/core/listener/QueryInsightsListener.java

ansjcy · 2025-04-28T23:00:11Z

src/main/java/org/opensearch/plugin/insights/core/listener/QueryInsightsListener.java


+    private void setExcludedIndices(List<String> excludedIndices) {
+        this.excludedIndicesPattern = excludedIndices.stream()
+            .map(index -> index.contains("*") ? index.replace("*", ".*") : index)


This is a little bit hacky, can we use org.opensearch.common.Strings.simpleMatch? It is designed for this type of matching.

Hi @ansjcy I could not find any method with simpleMatch or pattern inside org.opensearch.core.common.Strings

ansjcy · 2025-04-28T23:01:14Z

src/main/java/org/opensearch/plugin/insights/core/listener/QueryInsightsListener.java

+     * @param excludedIndices list of index to validate
+     */
+    public void validateExcludedIndices(@NonNull List<String> excludedIndices) {
+        for (String index : excludedIndices) {


We should use org.opensearch.common.Strings.isSimpleMatchPattern to validate the pattern

quangdutran · 2025-04-29T09:16:00Z

Hi @ansjcy integration has been added, could you take a look? Thanks, and please check the Strings class you mentioned, I don't think I could find the methods that you're referring to.

deshsidd · 2025-05-01T18:30:26Z

@quangdutran Builds failing and code hygiene checks failing. Can you please take a look and fix.

ansjcy · 2025-05-01T19:38:46Z

Builds failing and code hygiene checks failing

The failing is caused by security plugin not ready in 3.1. I believe it is ready now. We can do a rebase on main and the pipeline should pass.

ansjcy · 2025-05-01T19:42:27Z

@quangdutran my bad, it should be Regex.simpleMatch, see:
https://github.com/opensearch-project/OpenSearch/blob/987f4fa68273cec5096e04f51353b4842227eb58/server/src/main/java/org/opensearch/common/regex/Regex.java#L107

Will this function satisfy your need?

dzane17 · 2025-05-01T21:05:16Z

@ansjcy We were originally using Regex.simpleMatch, but there is some extra latency with this option. Every time simpleMatch is called, the provided index pattern string is converted to a pattern object. Instead of storing the index patterns as String we are precompiling them and storing as Pattern objects.

Previous thread and small benchmark here -> #308 (comment)

Signed-off-by: Du Tran <quangdutran809@gmail.com>

…tor UT Signed-off-by: Du Tran <quangdutran809@gmail.com>

… case Signed-off-by: Du Tran <quangdutran809@gmail.com>

quangdutran · 2025-05-04T09:25:25Z

@dzane17 Hi David, could you pls suggest me the solution for this:
My integr test TopQueriesRestIT.testExcludedIndices is failing when it runs with other tests because the top query counts the queries of other test as well. Is there any way I can somehow isolate my test from other tests?

ansjcy · 2025-05-05T20:42:20Z

but there is some extra latency with this option

Oh! Sorry I missed that. It makes sense to me now.

TopQueriesRestIT.testExcludedIndices is failing when it runs with other tests because the top query counts the queries of other test as well.

@quangdutran If we disable and reenable all the features before your test is run, ideally you can get a clean env without other tests runs, see below for references:

query-insights/src/test/java/org/opensearch/plugin/insights/core/service/grouper/MinMaxQueryGrouperBySimilarityIT.java

Line 30 in 16be187

waitForEmptyTopQueriesResponse();

,

query-insights/src/test/java/org/opensearch/plugin/insights/QueryInsightsRestTestCase.java

Line 198 in 16be187

protected String disableTopQueriesSettings() {

Another better option is, instead of asserting the number of top queries after the exclude settings, you can check that the returned SearchQueryRecord includes/ doesn't include the record with your included/excluded indices.

Signed-off-by: Du Tran <quangdutran809@gmail.com>

Signed-off-by: Du Tran <quangdutran809@gmail.com> (cherry picked from commit aea9b41) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

(cherry picked from commit aea9b41) Signed-off-by: Du Tran <quangdutran809@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

quangdutran mentioned this pull request Apr 17, 2025

[FEATURE] Exclude certain indices from top N queries #260

Closed

dzane17 reviewed Apr 17, 2025

View reviewed changes

quangdutran force-pushed the feat-260-exclude-indices branch from 4c0b18b to 0ec43d2 Compare April 20, 2025 13:17

quangdutran marked this pull request as ready for review April 20, 2025 13:20

quangdutran requested review from ansjcy, deshsidd and jainankitk as code owners April 20, 2025 13:20

quangdutran force-pushed the feat-260-exclude-indices branch from 0ec43d2 to 374929f Compare April 20, 2025 13:31

dzane17 reviewed Apr 21, 2025

View reviewed changes

deshsidd reviewed Apr 21, 2025

View reviewed changes

quangdutran requested review from deshsidd and dzane17 April 22, 2025 08:21

dzane17 reviewed Apr 22, 2025

View reviewed changes

quangdutran requested a review from dzane17 April 23, 2025 07:14

dzane17 reviewed Apr 23, 2025

View reviewed changes

dzane17 approved these changes Apr 28, 2025

View reviewed changes

ansjcy reviewed Apr 28, 2025

View reviewed changes

quangdutran added 4 commits May 4, 2025 10:42

Add setting to exclude certain indices from insight query

bc7afa2

Signed-off-by: Du Tran <quangdutran809@gmail.com>

Move validation logic of excluded index into listener class and refac…

1d7f8bf

…tor UT Signed-off-by: Du Tran <quangdutran809@gmail.com>

Move validation logic of excluded index into listener class and refac…

a8e1cc2

…tor UT Signed-off-by: Du Tran <quangdutran809@gmail.com>

Precompute the excluded indices pattern and validate index name upper…

e25db4d

… case Signed-off-by: Du Tran <quangdutran809@gmail.com>

quangdutran force-pushed the feat-260-exclude-indices branch from ed9077b to d642a6f Compare May 4, 2025 07:42

quangdutran force-pushed the feat-260-exclude-indices branch from d642a6f to 175de8e Compare May 4, 2025 08:05

quangdutran force-pushed the feat-260-exclude-indices branch from 175de8e to 94cfdb5 Compare May 6, 2025 06:59

Adding integration tests for excluded indices

b17ac8d

Signed-off-by: Du Tran <quangdutran809@gmail.com>

quangdutran force-pushed the feat-260-exclude-indices branch from 94cfdb5 to b17ac8d Compare May 6, 2025 08:27

quangdutran requested a review from ansjcy May 6, 2025 12:38

dzane17 merged commit aea9b41 into opensearch-project:main May 16, 2025
15 checks passed

dzane17 mentioned this pull request May 16, 2025

[DOC] Query Insights excluded_indices setting opensearch-project/documentation-website#9914

Closed

4 tasks

ansjcy added the backport 3.0 label Jun 9, 2025

opensearch-trigger-bot bot mentioned this pull request Jun 9, 2025

[Backport 3.0] Add setting to exclude certain indices from insight query #360

Merged

Add setting to exclude certain indices from insight query #308

Add setting to exclude certain indices from insight query #308

Uh oh!

Conversation

quangdutran commented Apr 17, 2025

Description

Issues Resolved

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dzane17 Apr 23, 2025

Choose a reason for hiding this comment

Uh oh!

quangdutran Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

quangdutran Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dzane17 Apr 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

deshsidd commented Apr 23, 2025

Uh oh!

dzane17 commented Apr 25, 2025

Uh oh!

quangdutran commented Apr 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Uh oh!

ansjcy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ansjcy Apr 28, 2025

Choose a reason for hiding this comment

Uh oh!

quangdutran Apr 29, 2025

Choose a reason for hiding this comment

Uh oh!

ansjcy Apr 28, 2025

Choose a reason for hiding this comment

Uh oh!

quangdutran commented Apr 29, 2025

Uh oh!

deshsidd commented May 1, 2025

Uh oh!

ansjcy commented May 1, 2025

Uh oh!

ansjcy commented May 1, 2025

Uh oh!

dzane17 commented May 1, 2025

Uh oh!

quangdutran commented May 4, 2025

Uh oh!

ansjcy commented May 5, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

quangdutran Apr 24, 2025 •

edited

Loading

quangdutran Apr 24, 2025 •

edited

Loading

quangdutran commented Apr 27, 2025 •

edited

Loading