[BUG] SnapshotUtils has inconsistent behavior when selectedIndices has negative patterns

[cwperks]

Describe the bug

The issue was uncovered in this issue in the security repo: opensearch-project/security#1652

and thoroughly described in these comments:

• [BUG] Restoring snapshot: indices exclusion triggers security_exception (creating OK, listing OK) security#1652 (comment)
• [BUG] Restoring snapshot: indices exclusion triggers security_exception (creating OK, listing OK) security#1652 (comment)

SecurityUtils.filterIndices (https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/snapshots/SnapshotUtils.java#LL68C86-L68C101) is used by the security plugin during snapshot restore requests to resolve to indices to restore from a snapshot. It was recently discovered that this method has inconsistent behavior based on the ordering of the indices in the list.

Consider a snapshot that contains:

.opendistro_security
.kibana_1
opensearch_dashboards_sample_data_flights
security-auditlog-2022.12.22

A request with the first pattern in .indices being a negative pattern:

// -.opendistro_security in the front of the list
POST _snapshot/my-first-repo/20221222-snapshot-all/_restore
{
    "indices": ["-.opendistro_security", "open*", "sec*"],
    "include_global_state" : false
}
// Restoring indices: [opensearch_dashboards_sample_data_flights, security-auditlog-2022.12.22, .kibana_1]

In this request, the negative pattern is now second, but the list contains the same three patterns.

// -.opendistro_security in the middle of the list
POST _snapshot/my-first-repo/20221222-snapshot-all/_restore
{
    "indices": ["open*", "-.opendistro_security", "sec*"],
    "include_global_state" : false
}

// Restoring indices: [opensearch_dashboards_sample_data_flights, security-auditlog-2022.12.22]

I would expect that these requests resolve to the same set of indices.

The issue is in SnapshotUtils here (https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/snapshots/SnapshotUtils.java#L91-L95):

...else if (indexOrPattern.charAt(0) == '-') {
   // if its the first, fill it with all the indices...
   if (i == 0) {
       result = new HashSet<>(availableIndices);
   }
   ...

It is adding all available indices to the result dataset if a negative pattern is first in the list.

That means that in the following scenario:

selectedIndices = ['-foo-2', 'foo-*'] that this will in effect resolve to all indices except for `-foo-2`. 

I think the intended behavior should be to resolve positive patterns first and then evaluate negative patterns.

If the list only contains negative patterns then it should be interpreted as all indices except for the ones in the list.

The behavior of this method should be consistent regardless of the order of the indices in selectedIndices

[stephen-crawford]

Resolved with #5626