Pattern exclusion not working properly on aliases when security is enabled

[costin]

Elasticsearch version (bin/elasticsearch --version):
master

Plugins installed: []

JVM version (java -version):
Java 10.0.1+10

OS version (uname -a if on a Unix-like system):
Win 10

Description of the problem including expected versus actual behavior:

When security is enabled, pattern exclusion is not enforced for aliases either for GetAliases or GetIndex with new IndicesOptions(EnumSet.of(Option.IGNORE_ALIASES), EnumSet.of(WildcardStates.OPEN))

Without security these calls work properly (see #33395, thanks to @original-brownbear ).

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.

1. Create two indices and aliases that start with the same prefix

PUT test_index1
{
    "aliases" : {
        "test_alias1" : {},
        "test_alias2" : {}
        }
}

PUT test_index2
{
    "aliases" : {
        "test_alias1" : {},
        "test_alias2" : {}
        }
}

PUT test_index_noalias
{
}

GET /_cat/aliases?v

alias                  index                     filter routing.index routing.search
test_alias1         test_index2            -      -             -
test_alias2         test_index2            -      -             -
test_alias1         test_index1            -      -             -
test_alias2         test_index1            -      -             -

2. GetAlias doesn't filter things

The following request:

 GetAliasesRequest aliasRequest = new GetAliasesRequest()
                    .local(true)
                    .aliases("test_*")
                    .aliases("-test_a*")
                    .indicesOptions(IndicesOptions.lenientExpandOpen());
   

Returns all aliases without excluding any.

3. GetIndex options not being applied:

IndicesOptions INDICES_ONLY_OPTIONS = new IndicesOptions(
            EnumSet.of(Option.ALLOW_NO_INDICES, Option.IGNORE_UNAVAILABLE, Option.IGNORE_ALIASES), EnumSet.of(WildcardStates.OPEN));

GetIndexRequest indexRequest = new GetIndexRequest()
                    .local(true)
                    .indices("test_*")
                    .indices("-test_a*")
                    .features(Feature.SETTINGS)
                    .includeDefaults(false)
                    .indicesOptions(INDICES_ONLY_OPTIONS);
  

Aliases are being returned (meaning both the IndicesOptions and the pattern exclusion were not applied).

Provide logs (if relevant):

[elasticmachine]

Pinging @elastic/es-security

[albertzaharovits]

I have a WIP branch https://github.com/albertzaharovits/elasticsearch/tree/fix-get-alias but I have to go in vacation, I'll resume next Tuesday.

Testwise I think https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolverTests.java should try to verify that whatever patterns it is resolving it conforms to how https://github.com/elastic/elasticsearch/blob/master/server/src/main/java/org/elasticsearch/cluster/metadata/MetaData.java resolves.

I am going to close this because I will treat it under the #29915 umbrella .

[albertzaharovits]

@costin I think I finally have a solid grasp of the issue.
First there is no security vulnerability. Under all circumstances wildcards are expanded by the security plugin in the limited visibility scope of the user. I was originally fooled by explicitly requested aliases (non wildcards) that were out of the user's visibility, but those requests are truthfully denied later in the process.

I have investigated the GetAlias API using wildcard (exclusion) expressions. There were indeed several problems.
The root of the problem is described in #34144 which has been merged and the broken behavior you experienced when the Security plugin was enabled is fixed.
However, GetAlias is broken in other ways, which has nothing to do with the Security plugin.
For example, the result of your GetAlias request:

curl -u elastic-admin:elastic-password -X GET "localhost:9200/_alias/test_*,-test_a*?pretty"
{
  "error" : "aliases [-test_a*,test_*] missing",
  "status" : 404
}

This shows no results, as expected, but there is an odd error being reported.
Again, this is a problem in ES core.
I have raised #34230 to fix it, but the fix elicits a bwc breaking change explained here #30536 (comment) . This needs some advertising to get it approved for merging. Any support on the discuss issue #30536 is welcome and appreciated!
There is another thing about wildcard expressions that might impact you, since you are using alias wildcards together with exclusions. It is described here #34291 . It's interesting because in this story the Security plugin is the hero not the villain!

Regarding the second part of your report, namely the GetIndex misbehavior, I was not able to reproduce it in any form and I am pretty sure the Security plugin is not breaking anything there. Also, it looks pretty serios, so I am thinking it's a setup problem somehow...
There are tests for these stuffs, maybe I am missing something in your setup/environment? I don't use the Transport client, I fake the request options in the code and issue curl calls.
Can you confirm if the problem persists, and if so, give some more reproduction steps?

[costin]

Thanks @albertzaharovits for the comprehensive reply. I'll take the changes for the spin and report back;
once thing to note from the start, there are differences in the rest vs java/transport API.
With GetAlias one can restrict the filtering only to aliases vs indices (GetAlias.setAliases() vs .setIndices()) while with GetIndexone can set specificIndexOptions- which SQL uses namely forGetIndex` to not expand to aliases but rather only indices.

Again, I'll pick the updates and report back.