[BUG][Concurrent Search] Search request for top anomalies from AD plugin is failing

[sohami]

Describe the bug

AD plugin exposes a top anomalies api which internally create a painless script based composite aggregation query. When concurrent search is enabled on a cluster, then the search request sometimes fails with error.

Ref:
Top Anomalies API: https://opensearch.org/docs/latest/observing-your-data/ad/api/#search-top-anomalies

Input to the API:

GET _plugins/_anomaly_detection/detectors/uBtS6HkBmDH36MzijITq/results/_topAnomalies
{
    "size": 10,
    "category_field": ["host", "service"],
    "order": "severity",
    "start_time_ms": 1622333056000,
    "end_time_ms": 1622937856000
}

Example Query which it creates:

"query": {
        "bool": {
            "filter": {
                "term": {
                    "detector_id": "uBtS6HkBmDH36MzijITq"
                }
            }
        }
    },
    "aggs": {
        "multi_buckets" : {
            "composite": {
                "sources": [
                    {
                        "host": {
                            "terms": {
                                "script": {
                                    "source": """
                                    String value = null;
                                    if (params == null || params._source == null || params._source.entity == null) {
                                        return "";
                                    }
                                    for (item in params._source.entity) {
                                        if (item['name'] == "host") {
                                            value = item['value'];
                                            break;
                                        }
                                    }
                                    return value;
                                    """,
                                    "lang": "painless"
                                }
                            }
                        }
                    },
                    {
                        "service": {
                            "terms": {
                                "script": {
                                    "source": """
                                    String value = null;
                                    if (params == null || params._source == null || params._source.entity == null) {
                                        return "";
                                    }
                                    for (item in params._source.entity) {
                                        if (item['name'] == "service") {
                                            value = item['value'];
                                            break;
                                        }
                                    }
                                    return value;
                                    """,
                                    "lang": "painless"
                                }
                            }
                        }
                    }
                ]
            },
            "aggregations": {
                "max": {
                    "max": {
                        "field": "anomaly_grade"
                    }
                },
                "multi_buckets_sort": {
                    "bucket_sort": {
                        "sort": [
                            { "max": { "order": "desc" } } 
                        ],
                        "size": 10                                
                    }
                }
            }
        }
    }

Related component

Search:Query Capabilities

To Reproduce

1. Create an index with AD plugin result index mapping. Ref here
2. Ingest the dummy data
3. Run the query shared in the description above.

Test failing in the plugin has sample of the index mapping for entity field and logic to create the data which can be used to reproduce. Ref here

Expected behavior

The query should be successful

Additional Details

Plugins
Anomaly Detection

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[sohami]

On looking more into this failure, there seems to be a race condition in the composite aggregation in general where the lookup instances is shared across slices. So this will have issues in all the composite aggregation path with concurrent segment search. We will need to disable the concurrent path with composite aggregation being used in the request to fall back to the non-concurrent path. The change will be 1 liner as we already have a mechanism to control it at per aggregation level.

[eirsep]

In the description, can you plz share the exception and stack trace related to the race condition